You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/06/17 01:49:49 UTC
[GitHub] incubator-metron pull request #158: METRON-219 Create Default Metron Dashboa...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/158
METRON-219 Create Default Metron Dashboard for Kibana 4
#### [METRON-219](https://issues.apache.org/jira/browse/METRON-219)
There is no sensible default dashboard that is created for Metron's Kibana 4 user interface. Create a default dashboard that is automatically deployed with Metron that has sensible defaults to enable the user to assess basic functioning of the environment.
#### Changes
- Enhanced Bro, Snort, and YAF Elasticsearch template definitions
- Created new Metron Dashboard in Kibana
- Automated the deployment of the Metron dashboard using Elasicdump
- Configured Kibana to land users on Metron dashboard by default
#### Testing
- Deployed successfully to Quick Dev Vagrant
- Deployed successfully to Amazon EC2
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-219
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/158.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #158
----
commit 9b196b59605cf09f71854f6ebd5ef0b23edeb11e
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-06-15T20:02:31Z
METRON-219 Enhanced elasticsearch indices for Bro, Snort, and YAF flows
commit c6ac1729a2495b0d9394cfbd822af2ca124c30bd
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-06-17T00:48:33Z
METRON-219 Automatic load of the .kibana index containing the Metron Dashboard
commit fb49ae2580e11cd656c5822b9a807efdacdf1352
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-06-17T01:10:09Z
METRON-219 Redirect users going to / to the Metron Dashboard by default
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/158
@dlyle65535 and I spent some time debugging. The `unsupported_operation_exception` coming from Kibana seems to be caused by the index templates not being loaded. The Kibana dashboard expects certain fields to be of certain types. Without the index templates, Elasticsearch likely chose a different type which caused the dashboard to blow up.
Now we need to figure out why Ansible did not create the index templates. Digging through my logs, Ansible reported this when attempting to create the index templates. This makes it seems like the templates exist, but clearly they were not created.
```
TASK [metron_elasticsearch_templates : Add Elasticsearch templates for topologies] ***
ok: [ec2-52-40-17-98.us-west-2.compute.amazonaws.com] => (item=/Users/nallen/Development/incubator-metron/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template)
ok: [ec2-52-40-17-98.us-west-2.compute.amazonaws.com] => (item=/Users/nallen/Development/incubator-metron/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/snort_index.template)
ok: [ec2-52-40-17-98.us-west-2.compute.amazonaws.com] => (item=/Users/nallen/Development/incubator-metron/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/158
In trying to replicate the issue @james-sirota is seeing, I ran it up on AWS and am seeing the following. Hmm...
```
Error: [unsupported_operation_exception] custom format isn't supported
at respond (http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/kibana.bundle.js?v=9892:76378:16)
at checkRespForFailure (http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/kibana.bundle.js?v=9892:76341:8)
at http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/kibana.bundle.js?v=9892:74959:8
at processQueue (http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/commons.bundle.js?v=9892:42360:29)
at http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/commons.bundle.js?v=9892:42376:28
at Scope.$eval (http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/commons.bundle.js?v=9892:43604:29)
at Scope.$digest (http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/commons.bundle.js?v=9892:43415:32)
at Scope.$apply (http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/commons.bundle.js?v=9892:43712:25)
at done (http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/commons.bundle.js?v=9892:38161:48)
at completeRequest (http://ec2-52-26-251-183.us-west-2.compute.amazonaws.com:5000/bundles/commons.bundle.js?v=9892:38359:8)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/158
When i spin this up in single node vagrant i have a standard Kibana interface come up. When I click on dashboard nothing is setup. When I try to load it there doesn't seem to be a way to load it either. Doesn't look like it's working
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/158
Sure. Added instructions to README.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #158: METRON-219 Create Default Metron Dashboa...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/158#discussion_r67512229
--- Diff: metron-deployment/roles/kibana/tasks/elasticdump.yml ---
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+- name: Setup Nodesource Yum Repository
+ shell: "curl --silent --location https://rpm.nodesource.com/setup_4.x | bash -"
+
--- End diff --
Yeah, I can fix that. I guess it will have to be too steps; a `get_url` followed by a `shell` if we do it that way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/158
I'm still +1 on this including this change, provided it worked on EC2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/158
Looks good, +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/158
I added some additional logic to validate that the index templates are created. If they are not, it will fail fast.
I have seen the index templates not be created. At the same time have seen everything work more often times than not. The template index creation logic was not changed in this PR, unless it is somehow impacted somehow by what is in the index template definition itself.
I'm comfortable pushing this in, if we have consensus from the larger community.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #158: METRON-219 Create Default Metron Dashboa...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/158#discussion_r67508498
--- Diff: metron-deployment/roles/kibana/tasks/elasticdump.yml ---
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+- name: Setup Nodesource Yum Repository
+ shell: "curl --silent --location https://rpm.nodesource.com/setup_4.x | bash -"
+
--- End diff --
Sorry, just noticed, this creates a warning ``` [WARNING]: Consider using get_url module rather than running curl``` when running Ansible. Not changing my vote, but if you happen to go back in here while fixing other issues, could you use get_url instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/158
Latest update: This works for me on Vagrant, but in AWS the Index Templates are lost when Elasticsearch is restarted towards the end of the deployment process. My hunch is that this is a pre-existing condition that we never noticed before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/158
got my +1 after that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/158
+1. Ran it up on vagrant and got it to work
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/158
+1, FWIW.
It is solid on vagrant. It seems to work mostly on AWS, but not all the time. I have added additional checks where possible. I'm going to recommend that we merge this, get more people using it, and then learn from their experience and patch if needed.
When it does not work, it is because the index templates do not exist when ES is restarted by Monit. If it does not work, here is the work around.
(1) Define which Elasticsearch master to work with.
```
export ES_HOST="http://ec2-52-25-237-20.us-west-2.compute.amazonaws.com:9200"
```
(2) Confirm the index templates are missing.
```
curl -s -XPOST $ES_HOST/_template/
```
(3) Manually load the index templates.
```
cd metron-deployment
export ES_HOST="http://ec2-52-25-237-20.us-west-2.compute.amazonaws.com:9200"
curl -s -XPOST $ES_HOST/_template/bro_index -d @roles/metron_elasticsearch_templates/files/es_templates/bro_index.template
curl -s -XPOST $ES_HOST/_template/snort_index -d @roles/metron_elasticsearch_templates/files/es_templates/snort_index.template
curl -s -XPOST $ES_HOST/_template/yaf_index -d @roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template
```
(4) Delete the existing indexes. Only a new index will use the template.
```
curl -s -XDELETE $ES_HOST/_template/yaf_index
curl -s -XDELETE $ES_HOST/_template/bro_index
curl -s -XDELETE $ES_HOST/_template/snort_index
```
(5) Open up Kibana and wait for the new indexes to be created. The dashboard should now work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/158
Those instructions in the situation where you have to adjust, can you add them to the aws instructions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #158: METRON-219 Create Default Metron Dashboa...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/158
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/158
+1. Ran it up in quick-dev-platform. Worked well, I like the look of it. I would like to understand what was going on whe @james-sirota ran it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---