You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2017/01/26 21:29:41 UTC
[GitHub] incubator-metron pull request #427: METRON-676 Create Zeppelin Notebook for ...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/427
METRON-676 Create Zeppelin Notebook for YAF Telemetry
Created a Zeppelin Notebook that serves as a basic template for a situational awareness dashboard for the YAF flow telemetry produced by Metron.
In practice, this notebook should be enhanced and customized to leverage enrichments specific to your production environment. The notebook provides a fair introduction into the mechanics of using Zeppelin/Spark to work with the telemetry that is archived by Metron in HDFS.
The Zeppelin Notebook is deployed with Metron through the MPack and can be installed by using the "Metron" > "Service Actions" > "Zeppelin Notebook Import" action in Ambari.
[METRON-676](https://issues.apache.org/jira/browse/METRON-676) contains a screen capture of the dashboard when it is run with roughly 7 days of archived telemetry data.
### Dependency
This change is dependent on #423 , which is why you will see those commits included here. Once #423 hits master, I will rebase on master.
### Testing
I tested this change by following these steps.
#### Build It
* Build Metron
```
cd incubator-metron
mvn clean package -DskipTests
```
* Start Docker on your build machine
* Build Metron RPMs
```
cd metron-deployment
mvn clean package -Pbuild-rpms -DskipTests
```
* Build Ambari MPack
```
cd metron-deployment
mvn clean package
```
#### Setup Test VM
* Launch Vagrant and install Ambari only
```
cd metron-deployment/vagrant/quick-dev-platform
vagrant --ansible-tags=ambari up
```
* Copy artifacts to VM
```
scp metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm vagrant@node1:/tmp
scp metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz vagrant@node1:/tmp
```
* Stage RPMS
```
vagrant ssh
sudo su -
mkdir /localrepo
cp /tmp/metron*.rpm /localrepo
```
#### Install Ambari MPack
* Install MPack
```
ambari-server install-mpack --mpack=/tmp/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
```
* Restart Ambari
```
service ambari-server restart
```
#### Deploy Metron with MPack
* You may need to clear the browser cache to see the additional options installed by the MPack.
* Login to Ambari at http://node1:8080
* Click "Actions" -> "Add Services", then choose "Metron"
#### Install Zeppelin Notebooks
* Login to Ambari at http://node1:8080.
* In Ambari, click "Metron" > "Service Actions" > "Zeppelin Notebook Import"
* Wait for the action to complete in Ambari.
* Login to Zeppelin at http://node1:9995
* Search for the notebook named "Metron - YAF Telemetry"
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-676
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/427.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #427
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #427: METRON-676 Create Zeppelin Notebook for ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/427
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #427: METRON-676 Create Zeppelin Notebook for YAF Tel...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/427
@nickwallen Would it be reasonable to throw links to the screenshots onto the original discuss thread?
I'm +1 (by inspection) on the actual implementation, but I'd really love to see other perspectives on this (and since I'm not an expert, I'm a little hesitant to just close this ticket out myself).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #427: METRON-676 Create Zeppelin Notebook for YAF Tel...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/427
Here are the links to screen captures of what the dashboard looks-like.
* [Report, read-only mode](https://issues.apache.org/jira/secure/attachment/12849721/Metron%20-%20YAF%20Telemetry%20-%20Report%20Mode.png)
* [Edit mode](https://issues.apache.org/jira/secure/attachment/12849578/Metron%20-%20YAF%20Telemetry.png)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #427: METRON-676 Create Zeppelin Notebook for YAF Tel...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/427
@nickwallen Thanks for closing off that discussion. This has percolated awhile, and I'm +1 on it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #427: METRON-676 Create Zeppelin Notebook for YAF Tel...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/427
@nickwallen #423 is in (and breaking the README here)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #427: METRON-676 Create Zeppelin Notebook for YAF Tel...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/427
+1 Great job
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #427: METRON-676 Create Zeppelin Notebook for YAF Tel...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/427
@justinleet Sorry, I originally responded to the dev list and not here. I sent a link to the PR and JIRA, mentioning the screenshots, to the original discussion thread.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---