You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/06/23 21:40:00 UTC
[GitHub] incubator-metron pull request #175: METRON-243 Ensure templates are installe...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/175
METRON-243 Ensure templates are installed on all search hosts
### [METRON-243](https://issues.apache.org/jira/browse/METRON-243)
I have a working theory on this one. I will need some additional testing, but I wanted to get some eyeballs on this to validate my thought process. I would love it if others were to run this through full AWS deployments and report their results.
#### Problem
Kibana reports this error when access the Metron Dashboard. Does not occur on Vagrant deployment. Occurs sometimes on AWS deployments.
```
[unsupported_operation_exception] custom format isn't supported
```
#### Root Cause
The dashboard expects certain data types for the Snort, Bro, and YAF indices. If the indices were created WITHOUT the index template definitions, the dashboard will throw this error.
In some cases, the index templates that appear to be properly installed during deployment, are later dropped when Elasticsearch is restarted by Monit. This causes Kibana to report this error. Now what might cause the index templates to go missing? Here are the steps that can cause this to occur.
* The 'elasticsearch' role installs Elasticsearch on each node in the 'search' host group. After installation, Elasticsearch is not yet started.
* In the next step, the 'metron_elasticsearch_templates' role uses Ansible's 'run_once' functionality. The effect of using 'run_once' is that Ansible chooses a single node in the 'search' host group; let's call this host ES1.
* Only on ES1, Elasticsearch is started, the index templates are installed, and then the index templates are validated. We effectively have a single node Elasticsearch cluster at this point.
* After all components have been installed, the 'monit-start' role ensures that all services are stopped across the cluster. At this point the single-node Elasticsearch cluster, which was left running, is shutdown.
* The 'monit-start' role then determines which services need started and starts them. At this point all of the Elasticsearch nodes are started which creates a multi-node Elasticsearch cluster.
* Here is the sticky wicket...
* If ES1 is elected master, then the index template that was previously configured is propagated to the other nodes in the cluster.
* If ES1 is not elected master, then the index template is forgotten as the master knows nothing about those index templates.
This explains why it occurs sometimes, but not all the time for AWS deployments.
This would also explain why the problem never occurs on Vagrant deployments.
#### Fix
This PR ensures that ALL Elasticsearch nodes are started when the index templates are configured. A side effect is that the index templates are installed multiple times, once per host in the cluster. This does not cause a problem necessarily, but is duplicative.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-243
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/175.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #175
----
commit bb1665fd7e8944687e8cfbc400a49c37b12a0c1b
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-06-23T21:25:48Z
METRON-243 Ensure templates are installed on all search hosts
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #175: METRON-243 Ensure templates are installed on al...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/175
+1 as well. Looks good, solid diagnosis work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #175: METRON-243 Ensure templates are installe...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/175
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #175: METRON-243 Ensure templates are installed on al...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/175
I'm happy with the change. It has been working for me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #175: METRON-243 Ensure templates are installed on al...
Posted by liqk01 <gi...@git.apache.org>.
Github user liqk01 commented on the issue:
https://github.com/apache/incubator-metron/pull/175
But i have single node only , and i donnot want to delete existing index ,then how do i resolve it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #175: METRON-243 Ensure templates are installed on al...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/175
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #175: METRON-243 Ensure templates are installed on al...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/175
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---