You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Anand Subramanian (JIRA)" <ji...@apache.org> on 2018/01/30 06:27:00 UTC
[jira] [Commented] (METRON-1369) squid events are not seen in
Alerts UI
[ https://issues.apache.org/jira/browse/METRON-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344563#comment-16344563 ]
Anand Subramanian commented on METRON-1369:
-------------------------------------------
In order for the squid events to show up in the Alerts UI, there needs to be a template added into Elasticsearch so that it gets applied dynamically as the data flows in.
The following needs to be done before ingesting into squid sensor:
{code:java}
curl -XPUT 'http://node1:9200/_template/squid_index' -d '
{
"template": "squid_index*",
"mappings": {
"squid_doc": {
"dynamic_templates": [
{
"geo_location_point": {
"match": "enrichments:geo:*:location_point",
"match_mapping_type": "*",
"mapping": {
"type": "geo_point"
}
}
},
{
"geo_country": {
"match": "enrichments:geo:*:country",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_city": {
"match": "enrichments:geo:*:city",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_location_id": {
"match": "enrichments:geo:*:locID",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_dma_code": {
"match": "enrichments:geo:*:dmaCode",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_postal_code": {
"match": "enrichments:geo:*:postalCode",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_latitude": {
"match": "enrichments:geo:*:latitude",
"match_mapping_type": "*",
"mapping": {
"type": "float"
}
}
},
{
"geo_longitude": {
"match": "enrichments:geo:*:longitude",
"match_mapping_type": "*",
"mapping": {
"type": "float"
}
}
},
{
"timestamps": {
"match": "*:ts",
"match_mapping_type": "*",
"mapping": {
"type": "date",
"format": "epoch_millis"
}
}
},
{
"threat_triage_score": {
"mapping": {
"type": "float"
},
"match": "threat:triage:*score",
"match_mapping_type": "*"
}
},
{
"threat_triage_reason": {
"mapping": {
"type": "text",
"fielddata": "true"
},
"match": "threat:triage:rules:*:reason",
"match_mapping_type": "*"
}
},
{
"threat_triage_name": {
"mapping": {
"type": "text",
"fielddata": "true"
},
"match": "threat:triage:rules:*:name",
"match_mapping_type": "*"
}
}
],
"properties": {
"timestamp": {
"type": "date",
"format": "epoch_millis"
},
"source:type": {
"type": "keyword"
},
"ip_dst_addr": {
"type": "ip"
},
"ip_dst_port": {
"type": "integer"
},
"ip_src_addr": {
"type": "ip"
},
"ip_src_port": {
"type": "integer"
},
"alert": {
"type": "nested"
},
"guid": {
"type": "keyword"
}
}
}
}
}
'{code}
Validate that the template loaded as expected by running:
{code:java}
curl -XGET 'http://node1:9200/_template/squid_index?pretty'{code}
[~mmiklavcic] has updated the Wiki instructions here to reflect the same. Thank you Mike!
[https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry]
> squid events are not seen in Alerts UI
> --------------------------------------
>
> Key: METRON-1369
> URL: https://issues.apache.org/jira/browse/METRON-1369
> Project: Metron
> Issue Type: Bug
> Reporter: Anand Subramanian
> Assignee: Nick Allen
> Priority: Major
>
> On a metron cluster, I have added the squid telemetry by following the [Wiki instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].
> I am able to see the squid indices present in elasticsearch, but not able to see them in the alerts UI. On further checking, I found out that this is due to the missing nested "alert" field in the index mappings--which is required for the Alerts UI to pick up the index.
> This was confirmed in the rest logs as well.
> {code}
> 17/12/18 10:39:40 ERROR dao.ElasticsearchRequestSubmitter: Shard search failure [1/1]; reason=IllegalStateException: [nested] failed to find nested object under path [alert], index=squid_index_2017.12.18.10, shard=0, status=BAD_REQUEST, nodeId=NOjEpjy0RxCV1ltcqp4Erw
> [squid_index_2017.12.18.10/a7-5TaroSRCHMrJnCWkYww] QueryShardException[failed to create query: {
> "constant_score" : {
> "filter" : {
> "bool" : {
> "must" : [
> {
> "bool" : {
> "should" : [
> {
> "query_string" : {
> "query" : "*",
> "fields" : [ ],
> "use_dis_max" : true,
> "tie_breaker" : 0.0,
> "default_operator" : "or",
> "auto_generate_phrase_queries" : false,
> "max_determinized_states" : 10000,
> "enable_position_increments" : true,
> "fuzziness" : "AUTO",
> "fuzzy_prefix_length" : 0,
> "fuzzy_max_expansions" : 50,
> "phrase_slop" : 0,
> "escape" : false,
> "split_on_whitespace" : true,
> "boost" : 1.0
> }
> },
> {
> "nested" : {
> "query" : {
> "query_string" : {
> "query" : "*",
> "fields" : [ ],
> "use_dis_max" : true,
> "tie_breaker" : 0.0,
> "default_operator" : "or",
> "auto_generate_phrase_queries" : false,
> "max_determinized_states" : 10000,
> "enable_position_increments" : true,
> "fuzziness" : "AUTO",
> "fuzzy_prefix_length" : 0,
> "fuzzy_max_expansions" : 50,
> "phrase_slop" : 0,
> "escape" : false,
> "split_on_whitespace" : true,
> "boost" : 1.0
> }
> },
> "path" : "alert",
> "ignore_unmapped" : false,
> "score_mode" : "none",
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> },
> {
> "bool" : {
> "should" : [
> {
> "term" : {
> "status" : {
> "value" : "active",
> "boost" : 1.0
> }
> }
> },
> {
> "bool" : {
> "must_not" : [
> {
> "exists" : {
> "field" : "status",
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> }
> ],
> "must_not" : [
> {
> "exists" : {
> "field" : "metaalerts",
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> },
> "boost" : 1.0
> }
> }]; nested: IllegalStateException[[nested] failed to find nested object under path [alert]];
> at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:329)
> at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312)
> at org.elasticsearch.search.SearchService.parseSource(SearchService.java:617)
> at org.elasticsearch.search.SearchService.createContext(SearchService.java:485)
> at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461)
> at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)
> at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340)
> at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:337)
> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1533)
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: [nested] failed to find nested object under path [alert]
> at org.elasticsearch.index.query.NestedQueryBuilder.doToQuery(NestedQueryBuilder.java:281)
> at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
> at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:444)
> at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:420)
> at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
> at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:444)
> at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:418)
> at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
> at org.elasticsearch.index.query.AbstractQueryBuilder.toFilter(AbstractQueryBuilder.java:118)
> at org.elasticsearch.index.query.ConstantScoreQueryBuilder.doToQuery(ConstantScoreQueryBuilder.java:146)
> at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
> at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$1(QueryShardContext.java:313)
> at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:325)
> ... 14 more
> {code}
> Now, I was trying to find a way to add this field to the mappings, so I can get the squid alerts listed. We have a documented process for including this filed in the [Elasticsearch README doc|https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch/README.md#using-metron-with-elasticsearch-2x], but this did not work for squid, since there is no template file present for squid. The following command returns empty:
> {code}
> curl -XGET 'http://<eshost>:9200/_template/squid*'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)