You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/11/06 15:28:00 UTC
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from
TransportClient to new Java REST API
[ https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676894#comment-16676894 ]
ASF GitHub Bot commented on METRON-1834:
----------------------------------------
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1242
Adapted from https://github.com/apache/metron/pull/840#issuecomment-349103038
# Test Script
Run full dev and verify you see data populating the alerts UI.
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).
# Preliminaries
Setup env vars. I like to do something like the following:
```
echo export METRON_HOST=node1 >> /root/.bashrc && \
echo export HDP_HOME=/usr/hdp/current >> /root/.bashrc && \
echo export KAFKA_HOME=/usr/hdp/current/kafka-broker >> /root/.bashrc && \
export SOLR_VERSION="6.6.2" && \
echo export SOLR_VERSION="$SOLR_VERSION" >> /root/.bashrc && \
echo export SOLR_HOME="/var/solr/solr-\${SOLR_VERSION}" >> /root/.bashrc && \
echo export ELASTIC_HOME="/usr/share/elasticsearch" >> /root/.bashrc && \
echo export KIBANA_HOME="/usr/share/kibana" >> /root/.bashrc && \
echo export ZOOKEEPER=\${METRON_HOST}:2181 >> /root/.bashrc && \
echo export BROKERLIST=\${METRON_HOST}:6667 >> /root/.bashrc && \
echo export STORM_UI=http://\${METRON_HOST}:8744 >> /root/.bashrc && \
echo export ELASTIC=http://\${METRON_HOST}:9200 >> /root/.bashrc && \
echo export ES_HOST=http://\${METRON_HOST}:9200 >> /root/.bashrc && \
echo export KIBANA=http://\${METRON_HOST}:5000 >> /root/.bashrc && \
export METRON_VERSION="0.6.1" && \
echo export METRON_VERSION="$METRON_VERSION" >> /root/.bashrc && \
echo export METRON_HOME="/usr/metron/\${METRON_VERSION}" >> /root/.bashrc && \
source /root/.bashrc
```
# Deploy the dummy parser
* Edit `$METRON_HOME/config/zookeeper/parsers/dummy.json`:
```
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"dummy"
}
```
* Create the dummy kafka topic:
`/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic dummy --partitions 1 --replication-factor 1`
* Persist config changes: `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z node1:2181`
* Start via `$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s dummy`
# Send dummy data through
* Edit `~/msg.json` with the following content:
```
{ "guid" : "guid0", "sensor.type" : "dummy", "timestamp" : 100 }
```
* Send `msg.json` through to kafka via `cat ~/msg.json | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic dummy`
* Validate data has been written to the index:
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty'
```
## Test Case: Update via patch
* Patch the message in ES and create a new field 'project' by executing
the following:
```
curl -u user:password -X PATCH --header 'Content-Type: application/json' --header 'Accept: */*' -d '{
"guid" : "guid0",
"sensorType" : "dummy",
"patch" : [
{
"op": "add"
, "path": "/project"
, "value": "metron"
}
]
}' 'http://node1:8082/api/v1/update/patch'
```
* Validate that the message has a field 'project':
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
"_source" : [ "project" ]
}
'
```
## Test Case: Update via replace
* Replace the message in ES and create a couple of modifications:
* new field `new_field` == "brand new"
* modified `timestamp` == 7
Execute the following:
```
curl -u user:password -X POST --header 'Content-Type: application/json' --header 'Accept: */*' -d '{
"guid" : "guid0",
"sensorType" : "dummy",
"replacement" : {
"source:type": "dummy",
"guid" : "guid0",
"new_field" : "brand new",
"timestamp" : 7
}
}' 'http://node1:8082/api/v1/update/replace'
```
* Validate that the message has a field 'new_field':
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
"_source" : [ "new_field", "timestamp" ]
}
'
```
## Meta Alerts Test
### Set Up Base Data
We're going to set up a bit of base data.
Retrieve the current list of indices so we know where to put our data
```
curl 'node1:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open snort_index_2017.09.06.14 1 0 130 0 180.9kb 180.9kb
green open bro_index_2017.09.06.14 1 0 160 0 564.3kb 564.3kb
green open .kibana 1 0 52 0 71.2kb 71.2kb
green open metaalert_index 1 0 6 0 62.3kb 62.3kb
```
In this case, we care about `snort_index_2017.09.06.14` and `metaalert_index`. To make our lives easier, we'll add a couple of stripped down messages to the our snort index (Make sure to sub in the correct index name):
```
curl -XPUT 'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_1?pretty' -H 'Content-Type: application/json' -d'
{
"msg": "snort test alert",
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000,
"threat:triage:rules:0:reason": null,
"threat:triage:score": 10,
"is_alert": "true",
"ip_src_port": "50187",
"guid": "snort_test_1"
}
'
```
```
curl -XPUT 'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_2?pretty' -H 'Content-Type: application/json' -d'
{
"msg": "snort test alert 2",
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000,
"threat:triage:rules:0:reason": null,
"threat:triage:score": 10,
"is_alert": "true",
"ip_src_port": "50187",
"guid": "snort_test_2"
}
'
```
### Create a Meta Alert
At this point, we'll group these alerts together. In Ambari, go to Metron -> Quick Links -> Swagger UI and go to the "Meta Alert Controller".
Click on the create endpoint, and we'll want to send this request (but with the appropriate index subbed in):
```
{
"groups": [
"group_one",
"group_two"
],
"guidToIndices": {
"snort_test_1":"snort_index_2017.09.06.14",
"snort_test_2":"snort_index_2017.09.06.14"
}
}
```
Validate that the metaalert was created and looks good:
```
curl 'node1:9200/m*/_search?pretty'
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "metaalerts",
"_type" : "metaalert_doc",
"_id" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"_score" : 1.0,
"_timestamp" : 1504725383439,
"_source" : {
"average" : 10.0,
"min" : 10.0,
"median" : 10.0,
"alert" : [ {
"msg" : "snort test alert",
"threat:triage:rules:0:reason" : null,
"ip_dst_port" : "8080",
"ethsrc" : "0A:00:27:00:00:00",
"threat:triage:score" : 10,
"is_alert" : "true",
"protocol" : "TCP",
"source:type" : "snort",
"ip_dst_addr" : "192.168.66.121",
"ip_src_port" : "50187",
"guid" : "snort_test_1",
"ip_src_addr" : "192.168.66.1",
"threat:triage:rules:0:score" : 10,
"timestamp" : 1504708744000
}, {
"msg" : "snort test alert 2",
"threat:triage:rules:0:reason" : null,
"ip_dst_port" : "8080",
"ethsrc" : "0A:00:27:00:00:00",
"threat:triage:score" : 10,
"is_alert" : "true",
"protocol" : "TCP",
"source:type" : "snort",
"ip_dst_addr" : "192.168.66.121",
"ip_src_port" : "50187",
"guid" : "snort_test_2",
"ip_src_addr" : "192.168.66.1",
"threat:triage:rules:0:score" : 10,
"timestamp" : 1504708744000
} ],
"max" : 10.0,
"threat:triage:score" : 20.0,
"count" : 2,
"guid" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"groups" : [ "group_one", "group_two" ],
"sum" : 20.0,
"status" : "active"
}
} ]
}
}
```
Note that both sub alerts are present, the various counts are filled in, a GUID has been given specifically to this meta alert, etc.
### Searching
Searching from the REST API works mostly as expected. After the above data has been created, use the search endpoint to run this query:
```
{
"from": 0,
"size": 2,
"indices": [
"*"
],
"query": "guid=snort_test_2",
"sort": [
{
"field": "_timestamp",
"sortOrder": "ASC"
}
]
}
```
The result should look similar to the following, and have the messages we'd expect (one from snort and one from metaalerts):
```
{
"total": 2,
"results": [
{
"id": "snort_test_2",
"source": {
"msg": "snort test alert 2",
"threat:triage:rules:0:reason": null,
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"threat:triage:score": 10,
"is_alert": "true",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_port": "50187",
"guid": "snort_test_2",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000
},
"score": 1,
"index": "snort_index_2017.09.06.14"
},
{
"id": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"source": {
"average": 10,
"min": 10,
"median": 10,
"alert": [
{
"msg": "snort test alert",
"threat:triage:rules:0:reason": null,
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"threat:triage:score": 10,
"is_alert": "true",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_port": "50187",
"guid": "snort_test_1",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000
},
{
"msg": "snort test alert 2",
"threat:triage:rules:0:reason": null,
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"threat:triage:score": 10,
"is_alert": "true",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_port": "50187",
"guid": "snort_test_2",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000
}
],
"max": 10,
"threat:triage:score": 20,
"count": 2,
"guid": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"groups": [
"group_one",
"group_two"
],
"sum": 20,
"status": "active"
},
"score": 1,
"index": "metaalerts"
}
]
}
```
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---------------------------------------------------------------
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
> Reporter: Michael Miklavcic
> Assignee: Michael Miklavcic
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)