You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by justinleet <gi...@git.apache.org> on 2017/01/25 15:20:13 UTC
[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform
GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/423
METRON-270: Add Zeppelin to the platform
Adds Zeppelin to the Ambari Management Pack portion. Adding to the Ansible / quick dev portion is happening a bit in parallel with Nick's work on METRON-346.
This ticket existed before the mpack, so if we're not comfortable having this just cover the mpack I'd prefer to split this ticket rather than force them together. This isn't packaging up any notebooks, so there's not deviation in functionality until notebooks are added.
Essentially, this ties into metron-indexing (because we rely on data generated there). I'm open to adjusting this if anybody feels strongly about it, but I think it's currently the best place. Zeppelin notebook JSON files are loaded from metron-platform/metron-indexing/src/main/config/zeppelin/. These files can be placed into subdirs for organization if desired. (e.g. zeppelin/bro/bro.json) and must end with .json.
A custom action is added to the mpack to import these notebooks. This action is available regardless of whether or not Metron itself is running. Zeppelin configuration is autopopulated by the management pack.
Zeppelin allows for duplicate notebook names (they'll be given differing IDs). I didn't implement a way to track installed notebooks, but this is potentially a good future feature (to allow us to delete all installed notebooks, etc.). Once the management pack installs the notebooks, they're treated as belonging to Zeppelin entirely and can be managed there.
For testing, I created a few notebooks (valid, invalid, misnamed and in subdirs), updated the spec files, recreated the RPMs, and ran this up on a pretty constrained local cluster. I was able to see them and run sections of them as appropriate.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/incubator-metron zeppelin_dashboard
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/423.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #423
----
commit e5b392cc73d49fa4d9a3420df556e480c8f918af
Author: justinjleet <ju...@gmail.com>
Date: 2017-01-23T13:36:24Z
Working initially
commit 1cafd1d7c03d9fc25d7d100ddb369bc43bec4aad
Author: justinjleet <ju...@gmail.com>
Date: 2017-01-24T20:46:24Z
Building directory for loading notebook files
commit 58f2ed354c86d66d75a850d97a3af38ad6fc4a53
Author: justinjleet <ju...@gmail.com>
Date: 2017-01-25T14:57:11Z
Updating READMEs
commit c7e3958d6c99d723d9010d804a10a38ffbd053e9
Author: justinjleet <ju...@gmail.com>
Date: 2017-01-25T15:11:10Z
Removing extraneous spec change from testing, and updating README
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #423: METRON-270: Add Zeppelin to the platform
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/423
I had the wrong directory. If I put a Notebook in `/usr/metron/0.3.0/config/zeppelin/` then it works properly. Nice!
```
2017-01-26 18:12:53,238 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.5.0.0-1245
2017-01-26 18:12:53,238 - Checking if need to create versioned conf dir /etc/hadoop/2.5.0.0-1245/0
2017-01-26 18:12:53,240 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-01-26 18:12:53,273 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 exist already', '')
2017-01-26 18:12:53,273 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-01-26 18:12:53,295 - checked_call returned (0, '')
2017-01-26 18:12:53,296 - Ensuring that hadoop has the correct symlink structure
2017-01-26 18:12:53,296 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-01-26 18:12:53,299 - Execute['curl -s -XPOST http://node1:9995/api/notebook/import -d "@/usr/metron/0.3.0/config/zeppelin/Metron - YAF Telemetry.json"'] {'logoutput': True}
{"status":"CREATED","message":"","body":"2C88YEF19"}
Command completed successfully!
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/423#discussion_r98042135
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -115,6 +116,16 @@ def elasticsearch_template_delete(self, env):
yaf_cmd = ambari_format('curl -s -XDELETE "http://{es_http_url}/yaf_index*"')
Execute(yaf_cmd, logoutput=True)
+ def zeppelin_dashboard_install(self, env):
+ from params import params
+ env.set_params(params)
+
+ for dirName, subdirList, files in os.walk(params.metron_config_zeppelin_path):
--- End diff --
I'll add logging for that. You're right, in the case that there are no notebooks, it'd be helpful to see what it's using.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/423#discussion_r98040142
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -115,6 +116,16 @@ def elasticsearch_template_delete(self, env):
yaf_cmd = ambari_format('curl -s -XDELETE "http://{es_http_url}/yaf_index*"')
Execute(yaf_cmd, logoutput=True)
+ def zeppelin_dashboard_install(self, env):
+ from params import params
+ env.set_params(params)
+
+ for dirName, subdirList, files in os.walk(params.metron_config_zeppelin_path):
--- End diff --
Would it make sense to log the path that it is looking for the Notebooks in; `params.metron_config_zeppelin_path`? I'd like to see this in the Ambari status output, especially if it doesn't find any Notebooks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/423#discussion_r98042802
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml ---
@@ -187,6 +187,14 @@
<timeout>600</timeout>
</commandScript>
</customCommand>
+ <customCommand>
+ <name>ZEPPELIN_DASHBOARD_INSTALL</name>
--- End diff --
Seems like they use "Import" instead of "Install". How about "Zeppelin Notebook Import"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/423
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/423#discussion_r98044156
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml ---
@@ -187,6 +187,14 @@
<timeout>600</timeout>
</commandScript>
</customCommand>
+ <customCommand>
+ <name>ZEPPELIN_DASHBOARD_INSTALL</name>
--- End diff --
Perfect.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #423: METRON-270: Add Zeppelin to the platform
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/423
So far, I have not been able to get the "Zeppelin Dashboard Install" mechanism to work.
I am not aware of step-by-step instructions to use the MPack, but maybe I am missing them. Thought it would be beneficial to document here since it took me a while to piece all this together. Here is exactly what I did. What did I do wrong?
### Build
* Build Metron
```
cd incubator-metron
mvn clean package -DskipTests
```
* Start Docker on your build machine
* Build Metron RPMs
```
mvn clean package -Dbuild-rpms -DskipTests
```
* Validate Metron RPMs
```
$ find ./ -name "*.rpm"
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-common-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-data-management-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-elasticsearch-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-enrichment-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-indexing-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-parsers-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-pcap-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-profiler-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-solr-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/SRPMS/metron-0.3.0-201701261504.src.rpm
```
* Build Ambari MPack
```
cd metron-deployment
mvn clean package
```
* Validate Ambari MPack
```
$ find . -name "*mpack*.tar.gz"
./packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz
```
### Create Test VM
* Launch Vagrant and install Ambari only
```
cd metron-deployment/vagrant/quick-dev
vagrant --ansible-tags=ambari up
```
* Copy Artifacts to VM
```
scp incubator-metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm vagrant@node1:/tmp
scp incubator-metron/metron-deploment/packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz vagrant@node1:/tmp
```
* Prepare RPMS
```
vagrant ssh
sudo su -
mkdir /localrepo
cp /tmp/metron*.rpm /localrepo
```
### Install Ambari MPack
* Install MPack
```
ambari-server install-mpack --mpack=/tmp/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
service ambari-server restart
```
### Deploy Metron with MPack
* You may need to clear the browser cache to see the additional options installed by the MPack.
* Login to Ambari at http://node1:8080
* Click Add Services
* Select "Metron" and then continue the install
* SUCCESS! Validated that I was prompted to install Zeppelin and Spark as dependencies
### Install Zeppelin Notebooks
* Create a Zeppelin notebook and drop it at `/usr/metron/0.3.0/zeppelin/`.
* In Ambari, Click Metron > Service Actions > Zeppelin Dashboard Install.
* This is what I see from the Ambari status window.
```
2017-01-26 16:45:29,926 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.5.0.0-1245
2017-01-26 16:45:29,926 - Checking if need to create versioned conf dir /etc/hadoop/2.5.0.0-1245/0
2017-01-26 16:45:29,927 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-01-26 16:45:29,947 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 exist already', '')
2017-01-26 16:45:29,948 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-01-26 16:45:29,968 - checked_call returned (0, '')
2017-01-26 16:45:29,972 - Ensuring that hadoop has the correct symlink structure
2017-01-26 16:45:29,972 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
Command completed successfully!
```
* FAILURE! I then go to Zeppelin at `http://node1:9995/` and search for the notebook. The Notebook does not seem to be installed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #423: METRON-270: Add Zeppelin to the platform
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on the issue:
https://github.com/apache/incubator-metron/pull/423
@nickwallen , glad you got it working. Just BTW, the [wiki page](https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install) recently posted by Dima Kovalyov (sorry, can't find the github uid) has fairly detailed step-by-step instructions for Ambari install, which it looks like you pretty well replicated. It will need updating after @dlyle65535 commits the stuff from METRON-609 and METRON-634 (see PR#425).
Would be great if you or @justinleet add the Zeppelin steps to that wiki page :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/423#discussion_r98040344
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml ---
@@ -187,6 +187,14 @@
<timeout>600</timeout>
</commandScript>
</customCommand>
+ <customCommand>
+ <name>ZEPPELIN_DASHBOARD_INSTALL</name>
--- End diff --
Should we call this "Zeppelin Notebook Install"? I **think** that's the preferred Zeppelin nomenclature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/423#discussion_r98066760
--- Diff: metron-deployment/README.md ---
@@ -133,6 +133,8 @@ Notably, the URL for the GeoIP database that is preloaded (and is prefilled by d
After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearch templates. Similar to this, a custom Kibana action to Load Template is available.
+Another custom action is available in Ambari to import Zeppelin dashboards.
--- End diff --
Should we link from this mention here to the more detailed information that you have in `metron-platform/metron-indexing/README.md`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---