You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by ottobackwards <gi...@git.apache.org> on 2018/11/13 17:45:22 UTC
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
GitHub user ottobackwards opened a pull request:
https://github.com/apache/metron/pull/1261
METRON-1860 [WIP] new developer option for ansible in docker to deploy to vagrant
The goal of this PR is to provide a new "full_dev" option for new and old users that does not require as much setup and version matching to try Metron's full dev environment.
Currently, the vagrant up command runs ansible locally, on the host machine, to build and deploy metron. This means that the user must not only have Vagrant, Virtual Box and Docker, but must also have all the tools necessary to build metron ( maven, java, c++ 11 etc ) and run ansible ( python and others ). It has been a common source of problems for new users to get started with Metron because of version or setup problems.
This PR introduces a new metron-deployment/development option which tries to address this problem, and make it possible for the user to only have Vagrant, VirtualBox and Docker ( along with a local copy of the source ) to be able to run full dev.
The new option starts the Vagrant VM, but does not run ansible in it. Instead it runs a docker container which contains all the tools/versions necessary, and that container is what runs ansible.
This is a WIP, I am looking for initial feedback.
##Testing
Have the correct versions of vagrant, virtual box and docker installed and running
```
cd $METRON_SRC_ROOT/metron-deployment/development/centos6_docker_build
./build_and_run.sh
```
Answer yes to building the vagrant box.
Answer yes to building the docker machine
Go grab a coffee.
The end result should be full dev running in the vagrant instance.
### Differences
- This does not support skip tags passed on the cli
- This does not support provision
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [-] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [-] Have you written or updated unit tests and or integration tests to verify your changes?
- [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron vagrant-docker
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1261.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1261
----
commit 1ac44c0d78b25113a5ebe807baccf160ef9af139
Author: Otto Fowler <ot...@...>
Date: 2018-11-09T03:51:56Z
Initial commit - This provides an environment where only vagrant and docker ( and a copy of the metron codebase ) are required to run the Metron
full-dev vm with it's default setup.
This is the initial work, there will be refactorings
commit f54cc493f8e045d5f0ca669258f56a5576b92ad1
Author: Otto Fowler <ot...@...>
Date: 2018-11-13T17:32:38Z
refactored locations
----
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
@mmiklavc I have it integrated into IntelliJ, though it isn't as obvious as java errors
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1261
Valiant effort @ottobackwards . I am just wondering how much easier this really makes it?
Have you thought about just publishing a Metron Demo image to Vagrant Cloud? Does that scratch the same itch?
* Everything is already pre-installed (removes one source of potential problems for new users.)
* All a user needs is Vagrant and VirtualBox.
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/metron/pull/1261
@mmiklavc probably at some point. I assume it would require some cleanup for various scripts at the same time.
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1261
hey @ottobackwards - I am still wrapping my head around this, but one small nit is that I don't like all the confirmation prompts. With the prompts I have to constantly check back to ensure it is not stuck on another prompt waiting for me to do something.
As much as possible it should just be fire-and-forget, so I can run it, work on something else, and hopefully come back some time later with a functioning Metron install.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236247445
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
--- End diff --
Why `#*=`?
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
It is possible to imagine a number of scenarios, including that, but also needing to build with new hadoop versions ( can't loose build from scratch ).
There are a number of things we can do down the road.
I think this work is going to help people enough in the near term to land it, while we discuss longer term refactoring and workflow.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236262354
--- Diff: metron-deployment/development/centos6_docker_build/ansible/inventory/group_vars/all ---
@@ -0,0 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# only need to build the RPMs for CentOS
+metron_build_packages_cmd: "shell cd {{ metron_build_dir }}/metron-deployment && mvn clean package -DskipTests -Pbuild-rpms"
--- End diff --
yes
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1261
How does the build time compare to the current approach (`mvn clean install -DskipTests`)? Does it change at all since it is now run in a Docker container?
How does the time it takes to get Full Dev running compare (whatever is equivalent to `vagrant up`)?
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236249808
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP = $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==================================================="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
--- End diff --
Shellcheck prefers `$(date)` ¯\_(ツ)_/¯
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236250204
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP = $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==================================================="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
+if [[ -z "${NODE1_IP}" ]]; then exit 1; fi
+echo "Using NODE1 IP ${NODE1_IP}"
+
+echo "===============Running Docker==============="
+docker run -it \
+ -v ${VAGRANT_PATH}/../../..:/root/metron \
+ -v ~/.m2:/root/.m2 \
+ -v ${VAGRANT_PATH}:/root/vagrant \
+ -v ${ANSIBLE_PATH}:/root/ansible_config \
+ -v ${VAGRANT_KEY_PATH}:/root/vagrant_key \
+ -v ${VAGRANT_PATH}/logs:/root/logs \
+ -e ANSIBLE_CONFIG='/root/ansible_config/ansible.cfg' \
+ -e ANSIBLE_LOG_PATH="/root/logs/${LOGNAME}" \
+ -e ANSIBLE_SKIP_TAGS=${A_SKIP_TAGS} \
--- End diff --
From shellcheck.
```
In build_and_run.sh line 130:
-v ${VAGRANT_PATH}/../../..:/root/metron \
^-- SC2086: Double quote to prevent globbing and word splitting.
In build_and_run.sh line 132:
-v ${VAGRANT_PATH}:/root/vagrant \
^-- SC2086: Double quote to prevent globbing and word splitting.
In build_and_run.sh line 133:
-v ${ANSIBLE_PATH}:/root/ansible_config \
^-- SC2086: Double quote to prevent globbing and word splitting.
In build_and_run.sh line 134:
-v ${VAGRANT_KEY_PATH}:/root/vagrant_key \
^-- SC2086: Double quote to prevent globbing and word splitting.
In build_and_run.sh line 135:
-v ${VAGRANT_PATH}/logs:/root/logs \
^-- SC2086: Double quote to prevent globbing and word splitting.
In build_and_run.sh line 138:
-e ANSIBLE_SKIP_TAGS=${A_SKIP_TAGS} \
^-- SC2086: Double quote to prevent globbing and word splitting.
```
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236252426
--- Diff: metron-deployment/development/centos6_docker_build/README.md ---
@@ -0,0 +1,89 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+Metron on CentOS 6 Built in Docker
+==================================
+
+This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running CentOS 6.
+It utilizes Vagrant for the virtual machine, and Docker for the build and deployment. Therefore lessens the burden on the user to have the correct versions of the build and deployment tools in order to try Metron.
+
+Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services.
+
+Getting Started
+---------------
+
+### Prerequisites
+
+The computer used to deploy Apache Metron will need to have the following components installed.
+
+ - [Docker](https://www.docker.com/community-edition)
+ - [Vagrant](https://www.vagrantup.com) 2.0+
+ - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager)
+ - [Virtualbox](https://virtualbox.org) 5.0+
+
+Running the following script can help validate whether you have all the prerequisites installed and running correctly.
+
+ ```
+ metron-deployment/scripts/platform-info.sh
+ ```
+
+#### How do I install these on MacOS?
+
+Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
+
+1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/).
+
+1. Run the following command in a terminal to install all of the required tools.
+
+ ```
+ brew cask install vagrant virtualbox docker
+ vagrant plugin install vagrant-hostmanager
+ open /Applications/Docker.app
+ ```
+
+### Deploy Metron
+
+1. Ensure that the Docker service is running.
+
+1. Deploy Metron
+
+ ```bash
+ cd metron-deployment/development/centos6_docker
--- End diff --
`s/centos6_docker/centos6_docker_build/`
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1261
In the interest of pure (and almost certainly incredibly ignorant) speculation, along the lines of the vagrant cloud, is it theoretically possible to save off a base image that has the Hadoop and non-Metron installs done? Then if said base image is missing (or possibly has changed properties, e.g. HDP version) rebuild it, and if found just build Metron and install mpack? Or alternatively, don't load the end result to Vagrant Cloud, just upload the base image we install on top of. Then only update it on a base image update.
What I'm getting at is that it would be nice to be able to build off of master, in a way that doesn't require an external dependency, and still lets us cache off the majority of the install (setting up HDP and Ambari). At that point, full dev would essentially be (for most builds) "build metron, build RPMs, run up image, do Metron install and setup". Which is probably 20 minutes faster and would make me a substantially happier person.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236250814
--- Diff: metron-deployment/development/centos6_docker_build/docker_run_ansible.sh ---
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+#
+# This script runs IN the docker container
+#
+
+cd /root/metron || exit 1
+
+# make sure we have the right c++ tools
+source /opt/rh/devtoolset-6/enable
--- End diff --
Consider adding `# shellcheck disable=SC1091` prior.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236251078
--- Diff: metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
--- End diff --
From shellcheck
```
In run_docker_only.sh line 26:
echo ${VAGRANT_KEY_PATH}
^-- SC2086: Double quote to prevent globbing and word splitting.
```
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1261
@ottobackwards - yeah, that's probably reasonable. The base HDP image create option would be an additional improvement to existing functionality.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236262632
--- Diff: metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
+
+# move over to the docker area
+cd ../docker || exit 1
+pwd
--- End diff --
oh, the pwd.... I guess not, this is a debugging leftover. I'll remove
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236292888
--- Diff: metron-deployment/development/centos6_docker_build/docker_run_ansible.sh ---
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+#
+# This script runs IN the docker container
+#
+
+cd /root/metron || exit 1
+
+# make sure we have the right c++ tools
+source /opt/rh/devtoolset-6/enable
+
+# give the option to skip building metron, in case they have already done so
+#read -p " build metron? [yN] " -n 1 -r
+#echo
+#if [[ $REPLY =~ ^[Yy]$ ]]; then
+# USE TAGS
+#fi
--- End diff --
yes
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236250996
--- Diff: metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
+
+# move over to the docker area
+cd ../docker || exit 1
+pwd
+
+echo "===============Running Docker==============="
+docker run -it \
+ -v ${VAGRANT_PATH}/../../..:/root/metron \
+ -v ~/.m2:/root/.m2 \
+ -v ${VAGRANT_PATH}:/root/vagrant \
+ -v ${ANSIBLE_PATH}:/root/ansible_config \
+ -v ${VAGRANT_KEY_PATH}:/root/vagrant_key \
--- End diff --
From shellcheck.
```
In run_docker_only.sh line 34:
-v ${VAGRANT_PATH}/../../..:/root/metron \
^-- SC2086: Double quote to prevent globbing and word splitting.
In run_docker_only.sh line 36:
-v ${VAGRANT_PATH}:/root/vagrant \
^-- SC2086: Double quote to prevent globbing and word splitting.
In run_docker_only.sh line 37:
-v ${ANSIBLE_PATH}:/root/ansible_config \
^-- SC2086: Double quote to prevent globbing and word splitting.
In run_docker_only.sh line 38:
-v ${VAGRANT_KEY_PATH}:/root/vagrant_key \
^-- SC2086: Double quote to prevent globbing and word splitting.
```
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236252613
--- Diff: metron-deployment/development/centos6_docker_build/README.md ---
@@ -0,0 +1,89 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+Metron on CentOS 6 Built in Docker
+==================================
+
+This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running CentOS 6.
+It utilizes Vagrant for the virtual machine, and Docker for the build and deployment. Therefore lessens the burden on the user to have the correct versions of the build and deployment tools in order to try Metron.
+
+Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services.
+
+Getting Started
+---------------
+
+### Prerequisites
+
+The computer used to deploy Apache Metron will need to have the following components installed.
+
+ - [Docker](https://www.docker.com/community-edition)
+ - [Vagrant](https://www.vagrantup.com) 2.0+
+ - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager)
+ - [Virtualbox](https://virtualbox.org) 5.0+
+
+Running the following script can help validate whether you have all the prerequisites installed and running correctly.
+
+ ```
+ metron-deployment/scripts/platform-info.sh
+ ```
+
+#### How do I install these on MacOS?
+
+Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
+
+1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/).
+
+1. Run the following command in a terminal to install all of the required tools.
+
+ ```
+ brew cask install vagrant virtualbox docker
+ vagrant plugin install vagrant-hostmanager
+ open /Applications/Docker.app
+ ```
+
+### Deploy Metron
+
+1. Ensure that the Docker service is running.
+
+1. Deploy Metron
+
+ ```bash
+ cd metron-deployment/development/centos6_docker
+ ./build_and_run.sh -h
+ ```
+ ```bash
+ usage: ./build_and_run.sh
+ --skip-vagrant-up skip vagrant up
+ --force-docker-build force build docker machine
+ --skip-tags='tag,tag2,tag3' the ansible skip tags
+ -h/--help Usage information.
+
+ example: to skip vagrant up and force docker build with two tags
+ build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'
--- End diff --
`s/-skip-vagrant-up/--skip-vagrant-up/`
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236251967
--- Diff: metron-deployment/development/docker/README.md ---
@@ -45,12 +45,4 @@ This provides a Docker Container containing all of the prerequisites required to
```
cd /root/metron
mvn clean package -DskipTests
- ```
--- End diff --
This should be reinstated.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236260538
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
--- End diff --
To make sure we consume the option and value
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1261
@ottobackwards thanks for the submission. Per recent community comments, it sounds like we could use some improvements to how our build/deploy dep versions interact with other tooling that may require other versions of things.
Is this the correct dependency listings for each host/container?
The Docker container will be pre-configured with:
- Java 8
- Ansible 2.4.0+
- Python 2.7
- Maven 3.3.9
- C++11 compliant compiler, like GCC
And the developer host machine would now need to manage versions of:
- Vagrant 2.0+
- Vagrant Hostmanager Plugin
- Virtualbox 5.0+
- Docker
The deployment goes through these steps:
1. You mount m2 repo and your code dir in the Docker instance.
2. build_and_run optionally spins up Vagrant in VirtualBox.
3. build_and_run optionally creates Docker instance with pre-reqs for building and deploying Metron.
4. build_and_run runs the build and then calls the Ansible deployment scripts from within Docker.
At the end, you have an ephemeral Docker instance + Vagrant instance that has the running Metron instance?
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
I think spellcheck is integrated into travis, so what we _can_ do is make shellcheck a build dependency and execute it through maven in the poms
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236256548
--- Diff: metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
+
+# move over to the docker area
+cd ../docker || exit 1
+pwd
--- End diff --
Is this still useful?
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236261928
--- Diff: metron-deployment/development/centos6_docker_build/docker_run_ansible.sh ---
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+#
+# This script runs IN the docker container
+#
+
+cd /root/metron || exit 1
+
+# make sure we have the right c++ tools
+source /opt/rh/devtoolset-6/enable
--- End diff --
ok
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236253176
--- Diff: metron-deployment/development/centos6_docker_build/README.md ---
@@ -0,0 +1,89 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+Metron on CentOS 6 Built in Docker
+==================================
+
+This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running CentOS 6.
+It utilizes Vagrant for the virtual machine, and Docker for the build and deployment. Therefore lessens the burden on the user to have the correct versions of the build and deployment tools in order to try Metron.
+
+Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services.
--- End diff --
I'd suggest at most 2 spaces between sentences, if not one.
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
@mmiklavc That is basically correct. Except that the ansible version is 2.5, since it only applies to this build, and allows for the yaml log formatting.
Also, in the latest version, the ansible once again does the clean and build as opposed to the script. I had a lot of problems getting the c++ picked up from ansible and moved the build out of it for the time, but the idea was always to have ansible run the metron_build, and that has returned.
The reasoning for the prompts to build the vagrant box and the docker ->
- if you are using this during development, IE> we are working ON ansible or ON docker, you may fail in the docker or ansible stage without modifying the vm, and thus not need to vagrant up again. Likewise, you may not need to rebuild the docker machine if you have not made changes, _or_ you may in fact need to. I added these flags as I developed.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236261064
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP = $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==================================================="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --
ok
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
If you create an issue for your vagrant base machine with our hadoop / ambari already in it, you can assign it to me. @justinleet
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236251269
--- Diff: metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
--- End diff --
Shellcheck prefers `$(pwd)` ¯_(ツ)_/¯
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236385492
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP = $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==================================================="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --
Fixed
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1261
@JonZeolla - would it make sense to add ShellCheck to Travis? Seems like a lot of potentially useful detail here.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236262118
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
--- End diff --
ok
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
We could also use more tags, for example I may want to skip building the java, but not skip building the RPMs. Think of a dev flow -> I make my change, run my local tests and want to spin up full dev. It is already built, but needs the rpms, I should be able to make ansible skip the compile/package of java and still do the rpms/debs
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236260621
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
--- End diff --
This is kind of boilerplate
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236249643
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP = $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==================================================="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --
`s/"^\s/"^\\s/`
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236335016
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP = $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==================================================="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --
Hmm, I successfully executed it locally. Did you do it on the awk? It should only get applied to the grep.
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
@nickwallen I did not think of that. I was improving the process that stands today. I think in a world where the posted image exists, we would still want the ability to try the latest ( to verify a fix pre-release etc ).
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1261
If we go this approach, why not just replace the existing "Full Dev" environments (both centos and ubuntu) rather than add new environments to support, test, and keep in-sync?
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
The base option is like quick dev, but quick dev also had metron installed, so starting it involved replacing the bits that where in there, we didn't maintain quick dev ansible w/o the metron stuff and 'other' ansible with the base.
I think that is a good idea, but for a follow on to this. This effort itself is or may be just a first step to a couple of things.
The base image would not replace this, but be the base image that this functionality uses.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236262449
--- Diff: metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
+
+# move over to the docker area
+cd ../docker || exit 1
+pwd
--- End diff --
Not sure what you mean
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/metron/pull/1261
If we'd want to replace full dev we would need to get skip tags passed in appropriately, I use that a lot. That said I'm not 100% that we need to do that all at once.
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
@nickwallen Done
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236301587
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP = $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==================================================="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --
this change breaks the statement. I disabled the warning.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236257483
--- Diff: metron-deployment/development/centos6_docker_build/docker_run_ansible.sh ---
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+#
+# This script runs IN the docker container
+#
+
+cd /root/metron || exit 1
+
+# make sure we have the right c++ tools
+source /opt/rh/devtoolset-6/enable
+
+# give the option to skip building metron, in case they have already done so
+#read -p " build metron? [yN] " -n 1 -r
+#echo
+#if [[ $REPLY =~ ^[Yy]$ ]]; then
+# USE TAGS
+#fi
--- End diff --
We should clean up lines 30-35.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236251472
--- Diff: metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
--- End diff --
```
In run_docker_only.sh line 23:
echo ${ANSIBLE_PATH}
^-- SC2086: Double quote to prevent globbing and word splitting.
```
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
@nickwallen That is an option, but not something I would pick as the goal from the outset if you know what I mean.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236375985
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP = $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==================================================="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag' ${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --
ahhh, yes, I did it for both.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236267508
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
--- End diff --
I would have thought this would only provide back the value, filtering out the option.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236253505
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
--- End diff --
`s/-skip-vagrant-up/--skip-vagrant-up/`
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236254476
--- Diff: metron-deployment/development/centos6_docker_build/ansible/inventory/group_vars/all ---
@@ -0,0 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# only need to build the RPMs for CentOS
+metron_build_packages_cmd: "shell cd {{ metron_build_dir }}/metron-deployment && mvn clean package -DskipTests -Pbuild-rpms"
+
+# ambari
+ambari_host: "{{ groups.ambari_master[0] }}"
+hdp_host_group: "{{ groups.ambari_slave }}"
+ambari_port: 8080
+ambari_user: admin
+ambari_password: admin
+cluster_type: single_node_vm
+ambari_server_mem: 512
+java_home: /usr/jdk64/jdk1.8.0_77
+
+# hbase
+pcap_hbase_table: pcap
+tracker_hbase_table: access_tracker
+threatintel_hbase_table: threatintel
+enrichment_hbase_table: enrichment
+
+# metron
+metron_version: 0.6.1
--- End diff --
Just bringing awareness that this may get changed to `0.7.0` prior to the next release. It's possible this could get missed depending on the order of operations.
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1261
@ottobackwards I like @justinleet's suggestion on having a base option. I'd imagine what we could have is a mechanism that allows you to build an image with just HDP of a specific version, and snapshot that if you like for general use, along with the option of just running against a standard image. I dunno, have we just re-hashed quickdev here? I can't recall specifically how or why it became problematic for us to maintain. It seems like there should be a path through this that allows us to not go stale, but also save the 20 min building the base installation, as Justin pointed out.
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/metron/pull/1261
I'm going to take a stab at a further look next week. For now I gave it a quick run-up and it was successful.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236261518
--- Diff: metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo " --skip-vagrant-up skip vagrant up"
+ echo " --force-docker-build force build docker machine"
+ echo " --skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo " -h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP = $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==================================================="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
--- End diff --
ok
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
@nickwallen, yeah, I did prompts as I went along debugging. I was thinking that folks may not like them.
I'll parameterize things.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236255405
--- Diff: metron-deployment/development/centos6_docker_build/ansible/inventory/group_vars/all ---
@@ -0,0 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# only need to build the RPMs for CentOS
+metron_build_packages_cmd: "shell cd {{ metron_build_dir }}/metron-deployment && mvn clean package -DskipTests -Pbuild-rpms"
--- End diff --
I see this is in our current ubuntu and centos `all` files, but what is shell, is this an ansible thing?
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
anyone have any ideas of the best way to time these things?
---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1261
The integration test failure has to do with the profiler tests and seem unrelated.
---