You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by dlyle65535 <gi...@git.apache.org> on 2016/06/06 23:21:57 UTC
[GitHub] incubator-metron pull request #145: METRON-212: Allow additional Elasticsear...
GitHub user dlyle65535 opened a pull request:
https://github.com/apache/incubator-metron/pull/145
METRON-212: Allow additional Elasticsearch templates to be loaded to the index
Tested with quick-dev-platform.
Currently this will allow the use of custom Elasticsearch templates by adding *.template files to the metron_streaming/files/es_templates directory. Deployment will pick them up and send them to the index using the basename without the .template extension as the template name.
I'm interested if others think that I should add an optional custom-defined directory name to pick up additional templates. Easy to do, but I wasn't sure it was needed.
This can be verified by running deployment and then running GET _template/* using your index-putter of choice (curl, sense, whatever). You should see a template that corresponds to each filename.
e.g. for Bro:
```json
{
"bro_index": {
"order": 0,
"template": "bro_index*",
"settings": {},
"mappings": {
"bro_doc": {
"_timestamp": {
"enabled": true
},
"properties": {
"enrichments:geo:ip_dst_addr:location_point": {
"type": "geo_point"
},
"timestamp": {
"format": "epoch_millis",
"type": "date"
}
}
}
},
"aliases": {}
}
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlyle65535/incubator-metron METRON-212
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/145.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #145
----
commit 8e0398d2eafea3b6e234bbfb8baf7f9521c9021a
Author: David Lyle <dl...@gmail.com>
Date: 2016-06-06T23:12:24Z
METRON-212: Allow additional Elasticsearch templates to be loaded to the index
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/145
What's your concern about starting Elasticsearch? The service module should support the environment. In the initial deploy path, ES will be installed but stopped when we execute this. For re-deployment of enrichments, ES will be running so nothing will happen when start is called. May be worth taking out the stop, but that could affect the Monit stuff on initial deployment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/145
I'm wondering where the best place is to store things like Elasticsearch template definitions. Ultimately, creating an effective Elasticsearch template is an important part of creating a parser. Therefore, it might make more sense for template definitions to be stored somewhere with the rest of the platform code under `metron-platform`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #145: METRON-212: Allow additional Elasticsear...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/145#discussion_r66069571
--- Diff: metron-deployment/roles/metron_streaming/tasks/main.yml ---
@@ -51,4 +51,8 @@
- include: hdfs_purge.yml
+- include: create_index_templates.yml
+ when: install_elasticsearch == True
--- End diff --
One way to do it that might result in some cleaner code in multiple places, is to get rid of the generic `search` host group. In its place, we define a `solr` host group when we want Solr installed and an `elasticsearch` host group when we want Elasticsearch installed.
You then extract this index creation functionality into a separate role `metron-elasticsearch` (or something like that). You can then target this functionality at hosts using the `elasticsearch` host group. This ensures it will not run when a cluster is using Solr.
This follows the pattern being used in METRON-211 where we have `metron-kafka` and `metron-hbase`. It also gives us greater granularity in targeting this functionality to hosts versus an uber-role like `metron-streaming` that does a ton of different things. Even though those different things are all related to the same end goal; deploying Metron streaming functionality.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #145: METRON-212: Allow additional Elasticsear...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/145#discussion_r66067963
--- Diff: metron-deployment/roles/metron_streaming/tasks/main.yml ---
@@ -51,4 +51,8 @@
- include: hdfs_purge.yml
+- include: create_index_templates.yml
+ when: install_elasticsearch == True
--- End diff --
Due to METRON-180 we no longer use `install_elasticsearch`. We will need to find another way to do this. There are 8 different ways to do it. Trying to think of the best way... hmmm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #145: METRON-212: Allow additional Elasticsear...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/145#discussion_r66069406
--- Diff: metron-deployment/roles/metron_streaming/tasks/main.yml ---
@@ -51,4 +51,8 @@
- include: hdfs_purge.yml
+- include: create_index_templates.yml
+ when: install_elasticsearch == True
--- End diff --
Yeah, I think I'll just pull it out to a separate roll and figure it out from there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/145
I like the METRON-211 style: `metron_kafka`, `metron_hbase`, `metron_elasticsearch`. This also more closely matches the existing `metron_streaming` role
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/145
Bump - anyone? Bueller?
@nickwallen - those changes you requested have been for a bit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/145
The problem was that the template load was attached to the 'enrichment' host group. It looks like you've already fixed that though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/145
+1 looks good
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/145
Hmm. Yes, I get the point you're making. I could live with that.
But I could also live with `metron_elasticsearch_templates`, `metron_hbase_tables`, and `metron_kafka_topics`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/145
So, specifically, you'd like to see elasticsearch_templates -> metron_elasticsearch_templates?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/145
I was thinking `metron_elasticsearch`. That way, if we have other ES related code we need for Metron, it also works in the same role.
I have some changes I can make in METRON-211 too better adhere to this convention too. Will propose those changes in that PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/145
It would be nice for the new role names in METRON-211 and METRON-212 to be consistent. Which naming convention makes more sense?
- METRON-211 convention -> metron_kafka, metron_hbase, metron_elasticsearch
- METRON-212 convention -> elasticsearch_templates, kafka_topics, hbase_tables
The reasons I like the METRON-211 convention slightly better is that:
1. The role names include `metron` indicating that the role specifically affects the deployment of core Metron.
2. The names center around a core Hadoop-y service so that, for example, if we need to add additional functionality that impacts Elasticsearch we can throw it into the same `metron_elasticsearch` role and the name still makes sense.
I could certainly be persuaded that METRON-212 is better though. If that is the case, I will update the names in METRON-211.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #145: METRON-212: Allow additional Elasticsear...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/145
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/145
So we'll have metron_elasticsearch and elasaticsearch roles?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/145
\U0001f44d
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/145
I think we might have to change this role so that it does not attempt to start or stop Elasticsearch itself. It can check if it can talk to Elasticsearch (which is a good check to have) but ultimately for this role to work in different kinds of environments, it can't try and start/stop Elasticsearch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/145
I don't really have a preference, What would you like them to be called?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---