You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by justinleet <gi...@git.apache.org> on 2016/09/21 13:13:58 UTC
[GitHub] incubator-metron pull request #266: METRON-427 Create Ambari Management Pack...
GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/266
METRON-427 Create Ambari Management Pack for Metron Installation
This is an initial cut at an MVP for the Ambari Management Pack. Most of the ground work was completed in other Jiras (detailed in METRON-427 description). This takes that work, adds the enrichment portion similiar to indexing, elasticsearch, etc., and bundles it in an mpack.
This includes end to end from creating appropriate Kafka topics, preloading a MySQL database with GeoIP data for enrichment, and making it available to an Elasticsearch and Kibana instance. ES is optional, and can be redirected to an outside instance.
The directory structuring changed to reflect the mpack MVP, and accordingly a lot of files got moved/renamed. For most of the ground work, there are various fixes and tweaks for consistency to each other and to fix bugs found during testing.
Detailed instructions are found in the metron-deployment README.md in the 'Ambari Management Pack' section. Essentially these steps are:
- Build the RPMs per the previously existing instructions (make sure to run the main build first).
- Build the mpack using the metron-deployment POM. From metron-deployment just run `mvn clean package -DskipTests`. This will produce an mpack tar.gz, which is what will be used in the installation to the Ambari server.
- Place the RPMs in /localrepo on the nodes. We don't have remote RPMs yet, so this is a necessary evil that will go away.
- Place the mpack file on the ambari-server node.
- Install the mpack with `ambari-server` install-mpack --mpack=<mpack_location> --verbose`
- Update Storm's configuration in Ambari for topology.classpath to /etc/hbase/conf:/etc/hadoop/conf. Ideally this will be automated after this MVP.
- Run through the normal Ambari install process. There are some caveats about this (colocation of Metron services isn't enforced, and Storm + Kafka + HBase client need to be available). More details are in the README, but just make sure Storm + Kafka + HBase client are available and that you aren't installing to a node with MySQL already installed. The MySQL issue is most likely on a single node install
- Use the action to install Elasticsearch templates where you can start / stop services.
This was tested on an AWS cluster, and on a set of Docker instances where a lot of components (e.g. MapReduce) disabled to avoid memory issues.
Steps used on Docker were (the server and agent images already have most of the unnecessary components disabled. Make sure to change jleet to your user home:
`docker-machine rm default
docker-machine create --driver virtualbox --virtualbox-disk-size "51200" --virtualbox-memory "8192" default
docker-machine start default
eval $(docker-machine env default)
git clone https://github.com/sequenceiq/docker-ambari
cd docker-ambari
. ambari-functions
export AMBARI_SERVER_IMAGE=dlyle65535/ambari-server:2.4.0.1-jdk8
export AMBARI_AGENT_IMAGE=dlyle65535/ambari-agent:2.4.0.1-jdk8
export DOCKER_OPTS="-v /Users/jleet:/home/host -v /Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch:/localrepo -v /Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/:/mpack"
(mac only) sudo route add -net 172.17.0.0/16 $(docker-machine ip default)
amb-start-cluster 5
amb-shell
blueprint add --url https://raw.githubusercontent.com/dlyle65535/ambari-rest-client/docker-metron-test/src/main/resources/blueprints/docker-metron
cluster build --blueprint docker-metron
cluster assign --host amb1.service.consul --hostGroup master_1
cluster assign --host amb2.service.consul --hostGroup master_2
cluster assign --host amb3.service.consul --hostGroup master_3
cluster assign --host amb4.service.consul --hostGroup master_4
cluster create
<WAIT FOR CLUSTER CREATION TO COMPLETE AND ^C TO EXIT>
docker exec -it amb-server bash
ambari-server install-mpack --mpack=/mpack/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
ambari-server restart
http://amb-server.service.consul:8080 - install Metron service on amb4.service.consul
es_url: can be invalid for test purpose otherwise, use authentic
One can docker it - docker run -d -p 9200:9200 -p 9300:9300 elasticsearch -Des.cluster.name=metron
Storm UI Server: amb1.service.consul:8744
`
Limitations of the MVP (also detailed in README)
- MySQL install should be optional (and allow for using an existing instance).
- MySQL should not be installed on a node already running a MySQL instance (e.g. an Ambari Server using MySQL as its database).
- There is currently no hosting for RPMs remotely. They will have to be built locally.
- Colocation of appropriate services should be enforced by Ambari. See [#Installing Management Pack] for more details.
- Storm's topology.classpath is not updated with the Metron service install and needs to be updated separately.
- Several configuration parameters used when installing the Metron service could (and should) be grabbed from Ambari. Install will require them to be manually entered.
- Need to handle upgrading Metron
- Different versions of CentOS apparently have different named for MySQL, so we may have to adjust there.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/incubator-metron mpack
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/266.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #266
----
commit d4b8ea8c148f1753932f799e3c553847cdb3b05c
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-05T13:23:12Z
sample
commit 0193f467f6b79a81479d30798525d981757d3146
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-09T22:14:10Z
add parser and common-services
commit 67096076717087126bdcf1e07d5e28d5b55a52e7
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-10T22:12:04Z
finish commands. refactor
commit 09dc18b9266702e90f4a7f02ea33edac31be0317
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-12T16:00:13Z
fixes
commit b447901c078c2b943608e2b5304fb87fb2aaabad
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-12T16:18:18Z
config fixes
commit 724070de4696059b98a358d585ff724b2f2226f9
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-16T14:10:27Z
incremental progress. quicklinks. kafka home
commit e83964b800875ea341407e58b921f2429d0c517d
Author: David Lyle <dl...@gmail.com>
Date: 2016-08-16T20:39:43Z
Initial working version with mpack build and separate profile (build-rpms) for building rpms.
commit edd7b60c0565ef725858aa32e299c1b20d1f481b
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-16T21:01:46Z
incremental fixes
commit 3172d9137b05f3f4c271f2a121fa4223606c8221
Author: justinjleet <ju...@gmail.com>
Date: 2016-08-17T12:14:36Z
WIP Indexing Service
commit 955680b20c6684c30631257a1068791a52e92a0a
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-17T21:09:44Z
check service configured on start. fix zk load. fix topology start
commit 310550291051f7d649f8f4a51a80ab722758837c
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-18T20:21:33Z
service check
commit 2a308376fb57ceaf2c875c0f0fb754c372d74f6f
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-18T21:40:12Z
trying to fix service check
commit e9abe68bba0b1b95302c99a29f5fb872dd60f1c0
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-19T15:25:07Z
cleanup/refactor. fix service check ordering
commit 8c1a093807c2ee18a6c9f315510bce2c2d0baf69
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-19T16:07:02Z
fix service check
commit ffae457b34e94f5fdbf22a638c62bf69d032057a
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-19T21:34:14Z
METRON-383 Ambari install for Metron Parser topologies.
commit d147e24097a76032b64f705456e6f3cfc335bff3
Author: justinjleet <ju...@gmail.com>
Date: 2016-08-17T12:50:06Z
Metron Indexing Service in Ambari
commit ae7a62af1ce17abc86897f26d05c4fb751095134
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-22T15:40:48Z
remove errant file
commit eb21e3651ec9f2a94f853684d94dd1d49fbbee2b
Author: David Lyle <dl...@gmail.com>
Date: 2016-08-25T14:46:35Z
Merge branch 'master' into AMBARI_MANAGEMENTPACK
commit 3d6e8faae5f3903986f51d162960dd3dae63f0f9
Author: David Lyle <dl...@gmail.com>
Date: 2016-08-25T17:34:19Z
Working ES and Kibana install.
commit ebe5539a4923da38f3285573bd4f26ed9a6f6ae8
Author: David Lyle <dl...@gmail.com>
Date: 2016-08-25T17:38:26Z
Merge branch 'METRON-383' of https://github.com/mmiklavc/incubator-metron into AMBARI_MANAGEMENTPACK
commit 32f7c63d17b0d62012d0e542cddfb9e81b0606c1
Author: David Lyle <dl...@gmail.com>
Date: 2016-08-25T18:19:50Z
Working parser integration
commit 8ab5c7b7d73cb6f98bb6957bb53e4ebfaaeab04d
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-05T13:23:12Z
sample
commit 2a6f97830f72151d3813ac0a41d04ef31410fe87
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-09T22:14:10Z
add parser and common-services
commit c2735510f1949b3682a0ecb8473d791c56e71315
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-10T22:12:04Z
finish commands. refactor
commit 959991b6a0497acd93422da4c5cc8b09a43cd48c
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-12T16:00:13Z
fixes
commit fe7f46822104d6b313ce80516916dcde9a0add4e
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-12T16:18:18Z
config fixes
commit 8bab89166952c736962fe07a7478829575a5fb61
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-16T14:10:27Z
incremental progress. quicklinks. kafka home
commit 43b6d04a8712c7332dd45116113b29ef61ebf8f9
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-16T21:01:46Z
incremental fixes
commit 800322dc948cffbaf03788efc8cf756d8b8cb8cf
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-17T21:09:44Z
check service configured on start. fix zk load. fix topology start
commit 0a723a458154e87f94210756cc3e2e49d3a805ad
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-08-18T20:21:33Z
service check
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #266: METRON-427 Create Ambari Management Pack for Me...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/266
+1 Lets get this checked in so we can get it into the next build and have more people kick the tires on it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #266: METRON-427 Create Ambari Management Pack...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/266
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #266: METRON-427 Create Ambari Management Pack for Me...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/266
+1 by inspection
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---