You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by justinleet <gi...@git.apache.org> on 2017/09/06 15:00:52 UTC
[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...
GitHub user justinleet opened a pull request:
https://github.com/apache/metron/pull/734
METRON-1158: Build backend for grouping alerts into meta alerts
## Contributor Comments
The motivation for this is also in the ticket, but the idea is that we want to be able to group alerts into a meta alert that lives and is queryable alongside the alerts from the standard sources. These would be manually created, particularly from the UI, as a result of investigations and slicing and dicing done there.
A REST API endpoint allows for the creation of a meta alert and another allows for retrieving all meta alerts associated with a given alert (by GUID). The first is obviously directly useful, and the second is more useful for testing and validation, but may be useful in other contexts (e.g. finding events that tend to get swept up into sets of investigations).
The various search endpoints should work as expected, and be able to query on the metaalerts index.
A test plan is incoming, and a couple more things need to be checked out a bit more thoroughly, but it should be pretty set for review. If anyone is interested, it should be easy enough to spin up, use the REST APIs (both the couple small meta alert ones along with the other search ones) and interrogation of ES to see what's going on. It's not completely transparent, because searches against the child alerts of meta alerts need to be done against the `alert.<field>`, e.g. ` "query": "alert.guid:35adafd5-ffd1-4b80-806b-91d336b87220 OR guid:35adafd5-ffd1-4b80-806b-91d336b87220"`
The basic test plan outline is going to be along the lines:
Spin up full dev
Create a couple meta alerts through the REST API.
Ensure they come back as expected from queries.
Ensure they link up appropriately as seen from the other endpoint.
Ensure the other endpoints still work as expected (e.g. search should still work, etc.)
Documentation also needs some updating and will be added (but I'll obviously take into account anything anybody wants as I write more of it).
Notes
* Given that these are manually created, a single index is used, rather than a timed index like for the standard alerts
* Creation of meta alerts takes a map from GUID -> Index. This is so we can do a multiget on the items, rather than trying to search through every index for them.
* A list of groups is provided as metadata for the UI to be able to reproduce whatever slicing and dicing was used to group these alerts into a metaalert. This could be extended to take arbitrary metadata, but that seems like followon. It's possible we may want to do something akin to https://github.com/apache/metron/pull/702, but I'm not familiar enough with it to say what the benefits of any synergy are.
* Unfortunately, sorting has to be done on a common field. This means the metadata index is required, at index time, to populate the common field. In our case, we use the standard threat score field. By default, the meta alerts will use the sum of the child alerts threat scores, but this can also be set via configuration to "median", "min", "max", "average", and "count". All values are calculated and stored in the event that they have other interest, but can't comingle with standard alerts.
* It's possible that scripted sorting could be used to get around that limitation, but that's a potentially much larger change that interferes with existing sorting and requires appropriate construction of the script to handle. Plus, scripting generally isn't enabled by default for security reasons, iirc.
* The templates have an "alert" nested field added to them. To my incredible annoyance, this version of elasticsearch will fail on any searches where "alert" is missing rather than just consider it not found. Even though it's in a "should" clause. This is improved on in 5.x, where there's a param to avoid failure. The fields should be removed again as part of that upgrade.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/metron meta_backend
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/734.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #734
----
commit 09c009d3530ead5d5ef7ac7d1ac4e57f39592658
Author: justinjleet <ju...@gmail.com>
Date: 2017-08-03T11:59:01Z
Meta Alerts WIP
commit 274a49f453cdccfce5251eb78aa2c51216b5faf2
Author: justinjleet <ju...@gmail.com>
Date: 2017-08-30T01:47:32Z
Trying to fix integration test
commit 0bdf67a3103a4a44b206c6ae4147738c807ba7ba
Author: justinjleet <ju...@gmail.com>
Date: 2017-08-30T11:21:29Z
test fixes
commit 8fa96167a271551e29f29a4d2e427e87151bf2a6
Author: justinjleet <ju...@gmail.com>
Date: 2017-08-30T11:47:36Z
Handling meta implementation in config
commit 4806f6ab0905b093b0e97b7a7892903a4dac4319
Author: justinjleet <ju...@gmail.com>
Date: 2017-09-06T01:27:07Z
Removing extraneous notes file and adding README section
commit 01cc5009304a653e39e32495ab32426c01193018
Author: justinjleet <ju...@gmail.com>
Date: 2017-09-06T02:05:10Z
Misc cleanup
commit 4705fe83790dcae80ea0d6270f1d2af2d2e0e4ac
Author: justinjleet <ju...@gmail.com>
Date: 2017-09-06T04:52:50Z
Fixes
commit ad4e76d91f18c1181abe3ad741fe52ae252ce68d
Author: justinjleet <ju...@gmail.com>
Date: 2017-09-06T05:28:33Z
Updating templates to resolve a problem I foresaw then forgot. Also passing through another function in meta dao
commit 5aec2fbae9034227bd0ea7c38f6d4c479dd0283a
Author: justinjleet <ju...@gmail.com>
Date: 2017-09-06T12:49:49Z
Adjusting sorting
commit 5d2ef582681ef3c5e2d041c2d7c2e6dc47ec47e3
Author: justinjleet <ju...@gmail.com>
Date: 2017-09-06T13:12:00Z
Updating mapping and index config
----
---
[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/734#discussion_r138046617
--- Diff: metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java ---
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.integration;
+
+import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.File;
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
+import org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao;
+import org.apache.metron.elasticsearch.dao.MetaAlertStatus;
+import org.apache.metron.elasticsearch.integration.components.ElasticSearchComponent;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.MetaAlertDao;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.ReplaceRequest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class ElasticsearchMetaAlertIntegrationTest {
+
+ private static final int MAX_RETRIES = 10;
+ private static final int SLEEP_MS = 500;
+ private static final String SENSOR_NAME = "test";
+ private static final String INDEX_DIR = "target/elasticsearch_meta";
+ private static final String DATE_FORMAT = "yyyy.MM.dd.HH";
+ private static final String INDEX =
+ SENSOR_NAME + "_index_" + new SimpleDateFormat(DATE_FORMAT).format(new Date());
+ private static final String NEW_FIELD = "new-field";
+
+ private static String metaMappingSource;
+ private static IndexDao esDao;
+ private static IndexDao metaDao;
+ private static ElasticSearchComponent es;
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ buildMetaMappingSource();
+ // setup the client
+ es = new ElasticSearchComponent.Builder()
--- End diff --
This actually is the metron-elasticsearch integration testing, so it's outside the scope of the REST layer. This is pretty much the pattern that exists within that module.
The `MetaAlertControllerIntegrationTest` does use the mock ES backend (`InMemoryMetaAlertDao` maintains the mock and delegates to it similar to how the actual DAOs work)
---
[GitHub] metron issue #734: METRON-1158: Build backend for grouping alerts into meta ...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/734
## Create Meta Alert
The first thing to do is to build and deploy full dev with this PR.
Afterwards, we're going to set up a bit of base data.
Retrieve the current list of indices so we know where to put our data
```
curl 'node1:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open snort_index_2017.09.06.14 1 0 130 0 180.9kb 180.9kb
green open bro_index_2017.09.06.14 1 0 160 0 564.3kb 564.3kb
green open .kibana 1 0 52 0 71.2kb 71.2kb
green open metaalerts 1 0 6 0 62.3kb 62.3kb
```
In this case, we care about `snort_index_2017.09.06.14` and `metaalerts`. To make our lives easier, we'll add a couple of stripped down messages to the our snort index (Make sure to sub in the correct index name):
```
curl -XPUT 'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_1?pretty' -H 'Content-Type: application/json' -d'
{
"msg": "snort test alert",
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000,
"threat:triage:rules:0:reason": null,
"threat:triage:score": 10,
"is_alert": "true",
"ip_src_port": "50187",
"guid": "snort_test_1"
}
'
```
```
curl -XPUT 'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_2?pretty' -H 'Content-Type: application/json' -d'
{
"msg": "snort test alert 2",
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000,
"threat:triage:rules:0:reason": null,
"threat:triage:score": 10,
"is_alert": "true",
"ip_src_port": "50187",
"guid": "snort_test_2"
}
'
```
At this point, we'll group these alerts together. In Ambari, go to Metron -> Quick Links -> Swagger UI and go to the "Meta Alert Controller".
Click on the create endpoint, and we'll want to send this request (but with the appropriate index subbed in):
```
{
"groups": [
"group_one",
"group_two"
],
"guidToIndices": {
"snort_test_1":"snort_index_2017.09.06.14",
"snort_test_2":"snort_index_2017.09.06.14"
}
}
```
Validate that the metaalert was created and looks good:
```
curl 'node1:9200/m*/_search?pretty'
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "metaalerts",
"_type" : "metaalert_doc",
"_id" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"_score" : 1.0,
"_timestamp" : 1504725383439,
"_source" : {
"average" : 10.0,
"min" : 10.0,
"median" : 10.0,
"alert" : [ {
"msg" : "snort test alert",
"threat:triage:rules:0:reason" : null,
"ip_dst_port" : "8080",
"ethsrc" : "0A:00:27:00:00:00",
"threat:triage:score" : 10,
"is_alert" : "true",
"protocol" : "TCP",
"source:type" : "snort",
"ip_dst_addr" : "192.168.66.121",
"ip_src_port" : "50187",
"guid" : "snort_test_1",
"ip_src_addr" : "192.168.66.1",
"threat:triage:rules:0:score" : 10,
"timestamp" : 1504708744000
}, {
"msg" : "snort test alert 2",
"threat:triage:rules:0:reason" : null,
"ip_dst_port" : "8080",
"ethsrc" : "0A:00:27:00:00:00",
"threat:triage:score" : 10,
"is_alert" : "true",
"protocol" : "TCP",
"source:type" : "snort",
"ip_dst_addr" : "192.168.66.121",
"ip_src_port" : "50187",
"guid" : "snort_test_2",
"ip_src_addr" : "192.168.66.1",
"threat:triage:rules:0:score" : 10,
"timestamp" : 1504708744000
} ],
"max" : 10.0,
"threat:triage:score" : 20.0,
"count" : 2,
"guid" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"groups" : [ "group_one", "group_two" ],
"sum" : 20.0,
"status" : "active"
}
} ]
}
}
```
Note that both sub alerts are present, the various counts are filled in, a GUID has been given specifically to this meta alert, etc.
## Searching
Searching from the REST API works mostly as expected. After the above data has been created, use the search endpoint to run this query:
```
{
"from": 0,
"size": 2,
"indices": [
"*"
],
"query": "guid=snort_test_2",
"sort": [
{
"field": "_timestamp",
"sortOrder": "ASC"
}
]
}
```
The result should look similar to the following, and have the messages we'd expect (one from snort and one from metaalerts):
```
{
"total": 2,
"results": [
{
"id": "snort_test_2",
"source": {
"msg": "snort test alert 2",
"threat:triage:rules:0:reason": null,
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"threat:triage:score": 10,
"is_alert": "true",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_port": "50187",
"guid": "snort_test_2",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000
},
"score": 1,
"index": "snort_index_2017.09.06.14"
},
{
"id": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"source": {
"average": 10,
"min": 10,
"median": 10,
"alert": [
{
"msg": "snort test alert",
"threat:triage:rules:0:reason": null,
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"threat:triage:score": 10,
"is_alert": "true",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_port": "50187",
"guid": "snort_test_1",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000
},
{
"msg": "snort test alert 2",
"threat:triage:rules:0:reason": null,
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"threat:triage:score": 10,
"is_alert": "true",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_port": "50187",
"guid": "snort_test_2",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000
}
],
"max": 10,
"threat:triage:score": 20,
"count": 2,
"guid": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"groups": [
"group_one",
"group_two"
],
"sum": 20,
"status": "active"
},
"score": 1,
"index": "metaalerts"
}
]
}
```
---
[GitHub] metron issue #734: METRON-1158: Build backend for grouping alerts into meta ...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/734
Great job on this. I spent a lot of time testing and had trouble finding anything wrong. The one thing I did notice was that the "timestamp" field was missing from documents in the metaalerts index. This is the field we commonly use for sorting so we might want to consider adding it or switching our default sort field to "_timestamp" instead.
---
[GitHub] metron issue #734: METRON-1158: Build backend for grouping alerts into meta ...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/734
I'm still +1
---
[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/734#discussion_r138064525
--- Diff: metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java ---
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.integration;
+
+import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.File;
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
+import org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao;
+import org.apache.metron.elasticsearch.dao.MetaAlertStatus;
+import org.apache.metron.elasticsearch.integration.components.ElasticSearchComponent;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.MetaAlertDao;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.ReplaceRequest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class ElasticsearchMetaAlertIntegrationTest {
+
+ private static final int MAX_RETRIES = 10;
+ private static final int SLEEP_MS = 500;
+ private static final String SENSOR_NAME = "test";
+ private static final String INDEX_DIR = "target/elasticsearch_meta";
+ private static final String DATE_FORMAT = "yyyy.MM.dd.HH";
+ private static final String INDEX =
+ SENSOR_NAME + "_index_" + new SimpleDateFormat(DATE_FORMAT).format(new Date());
+ private static final String NEW_FIELD = "new-field";
+
+ private static String metaMappingSource;
+ private static IndexDao esDao;
+ private static IndexDao metaDao;
+ private static ElasticSearchComponent es;
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ buildMetaMappingSource();
+ // setup the client
+ es = new ElasticSearchComponent.Builder()
--- End diff --
You are correct. I read this wrong and thought it was the REST controller test.
---
[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/734#discussion_r137874476
--- Diff: metron-interface/metron-rest/README.md ---
@@ -361,6 +363,21 @@ Request and Response objects are JSON formatted. The JSON schemas are available
* 200 - Returns sample message
* 404 - Either Kafka topic is missing or contains no messages
+### `POST /api/v1/metaalert/searchByAlert`
+ * Description: Searches meta alerts to find any containing an alert for the provided GUID
+ * Input:
+ * guid - GUID of the alert
+ * Returns:
+ * 200 - Returns the meta alerts associated with this alert
+ * 404 - Either Kafka topic is missing or contains no messages
--- End diff --
Is this a mistake?
---
[GitHub] metron issue #734: METRON-1158: Build backend for grouping alerts into meta ...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/734
@merrimanr I added the timestamp field in the meta alert create with current timestamp. Should take care of it lining up with the other sources.
---
[GitHub] metron issue #734: METRON-1158: Build backend for grouping alerts into meta ...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/734
Tested this again in full dev and now the default sort is working as expected (due to timestamp being added). +1 from me. Nice job.
---
[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/734
---
[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/734#discussion_r137874898
--- Diff: metron-interface/metron-rest/README.md ---
@@ -361,6 +363,21 @@ Request and Response objects are JSON formatted. The JSON schemas are available
* 200 - Returns sample message
* 404 - Either Kafka topic is missing or contains no messages
+### `POST /api/v1/metaalert/searchByAlert`
+ * Description: Searches meta alerts to find any containing an alert for the provided GUID
+ * Input:
+ * guid - GUID of the alert
+ * Returns:
+ * 200 - Returns the meta alerts associated with this alert
+ * 404 - Either Kafka topic is missing or contains no messages
--- End diff --
Absolutely a mistake. Thought I'd fixed that, but apparently not.
---
[GitHub] metron issue #734: METRON-1158: Build backend for grouping alerts into meta ...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/734
@merrimanr Are you still +1 after the most recent changes before I commit this?
---
[GitHub] metron issue #734: METRON-1158: Build backend for grouping alerts into meta ...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/734
@merrimanr I added in one fix for some typing on meta alert updates and also merged in master and the group stuff
---
[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/734#discussion_r137889738
--- Diff: metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java ---
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.integration;
+
+import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.File;
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
+import org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao;
+import org.apache.metron.elasticsearch.dao.MetaAlertStatus;
+import org.apache.metron.elasticsearch.integration.components.ElasticSearchComponent;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.MetaAlertDao;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.ReplaceRequest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class ElasticsearchMetaAlertIntegrationTest {
+
+ private static final int MAX_RETRIES = 10;
+ private static final int SLEEP_MS = 500;
+ private static final String SENSOR_NAME = "test";
+ private static final String INDEX_DIR = "target/elasticsearch_meta";
+ private static final String DATE_FORMAT = "yyyy.MM.dd.HH";
+ private static final String INDEX =
+ SENSOR_NAME + "_index_" + new SimpleDateFormat(DATE_FORMAT).format(new Date());
+ private static final String NEW_FIELD = "new-field";
+
+ private static String metaMappingSource;
+ private static IndexDao esDao;
+ private static IndexDao metaDao;
+ private static ElasticSearchComponent es;
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ buildMetaMappingSource();
+ // setup the client
+ es = new ElasticSearchComponent.Builder()
--- End diff --
You might want to consider moving this to TestConfig instead. The REST layer depends heavily on the InMemory components and reuses them across all the integration tests. You are breaking the pattern although I understand why because setting up InMemory components is usually done in @BeforeClass in other modules. This would allow other controller integration tests to leverage this infrastructure without having to do anything.
The SearchController uses a mock ES backend but, in my humble opinion, it's hard to maintain and is unnecessary. Would be nice to switch that to what you've done here someday.
---