You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Laurens Vets <la...@daemon.be> on 2017/10/04 20:38:11 UTC
SUM aggregator not working?
No idea whether it's a bug yet, I just need a 2nd set of eyes :)
This is my event as indexed in ES (Obviously some parts have been
obfuscated):
{
"_index": "cloudtrail_index_2017.10.04.19",
"_type": "cloudtrail_doc",
"_id": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
"_score": null,
"_timestamp": 1507143907108,
"_source": {
"eventID": "9e3d5468-2d97-4b9a-9821-5c61fec8c158",
"additionalEventData:MFAUsed": "No",
"adapter:stellaradapter:end:ts": "1507143907145",
"threatinteljoinbolt:joiner:ts": "1507143907153",
"eventVersion": "1.05",
"threat:triage:rules:0:comment": "Checks whether the field is_work
is true or false.",
"sourceIPAddress": "208.110.73.106",
"eventSource": "signin.amazonaws.com",
"enrichmentsplitterbolt:splitter:begin:ts": "1507143907143",
"enrichmentjoinbolt:joiner:ts": "1507143907147",
"additionalEventData:MobileVersion": "No",
"threat:triage:rules:0:name": "Not WORK",
"source:type": "cloudtrail",
"original_string":
"{\"eventVersion\":\"1.05\",\"userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AIDAI5ITCMVR3BQV5DUFW\",\"arn\":\"arn:aws:iam::<ACCOUNTID>:user/<EMAIL>\",\"accountId\":\"<ACCOUNTID>\",\"userName\":\"<EMAIL>\"},\"eventTime\":\"2017-10-04T18:57:31Z\",\"eventSource\":\"signin.amazonaws.com\",\"eventName\":\"ConsoleLogin\",\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"208.110.73.106\",\"userAgent\":\"Mozilla/5.0
(X11; Ubuntu; Linux x86_64; rv:56.0) Gecko/20100101
Firefox/56.0\",\"requestParameters\":null,\"responseElements\":{\"ConsoleLogin\":\"Success\"},\"additionalEventData\":{\"LoginTo\":\"https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true\",\"MobileVersion\":\"No\",\"MFAUsed\":\"No\"},\"eventID\":\"9e3d5468-2d97-4b9a-9821-5c61fec8c158\",\"eventType\":\"AwsConsoleSignIn\",\"recipientAccountId\":\"<ACCOUNTID>\"}",
"eventTime": "2017-10-04T18:57:31Z",
"eventName": "ConsoleLogin",
"recipientAccountId": "<ACCOUNTID>",
"userIdentity:principalId": "AIDAI5ITCMVR3BQV5DUFW",
"threatintelsplitterbolt:splitter:end:ts": "1507143907148",
"threat:triage:rules:0:score": 20,
"timestamp": 1507143907108,
"threat:triage:rules:0:reason": "208.110.73.106 is not an WORK
network!",
"awsRegion": "us-east-1",
"is_work": false,
"userIdentity:userName": "<EMAIL>",
"enrichmentsplitterbolt:splitter:end:ts": "1507143907143",
"threat:triage:score": 20,
"is_alert": "true",
"userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
Gecko/20100101 Firefox/56.0",
"adapter:stellaradapter:begin:ts": "1507143907145",
"eventType": "AwsConsoleSignIn",
"userIdentity:arn": "arn:aws:iam::<ACCOUNTID>:user/<EMAIL>",
"userIdentity:accountId": "<ACCOUNTID>",
"userIdentity:type": "IAMUser",
"threatintelsplitterbolt:splitter:begin:ts": "1507143907148",
"guid": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
"additionalEventData:LoginTo":
"https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true",
"responseElements:ConsoleLogin": "Success"
},
"fields": {
"adapter:stellaradapter:end:ts": [
1507143907145
],
"threatinteljoinbolt:joiner:ts": [
1507143907153
],
"enrichmentsplitterbolt:splitter:end:ts": [
1507143907143
],
"enrichmentsplitterbolt:splitter:begin:ts": [
1507143907143
],
"enrichmentjoinbolt:joiner:ts": [
1507143907147
],
"adapter:stellaradapter:begin:ts": [
1507143907145
],
"eventTime": [
1507143451000
],
"threatintelsplitterbolt:splitter:begin:ts": [
1507143907148
],
"threatintelsplitterbolt:splitter:end:ts": [
1507143907148
],
"timestamp": [
1507143907108
]
},
"sort": [
1507143451000
]
}
This is my sensor configuration:
{
"enrichment": {
"fieldMap": {
"stellar": {
"config": {
"is_work": "IN_SUBNET(if IS_IP(sourceIPAddress) then
sourceIPAddress else NULL, '1.2.3.4/16', '5.6.7.8/23')"
}
}
},
"fieldToTypeMap": {},
"config": {}
},
"threatIntel": {
"fieldMap": {
"stellar": {
"config": [
"is_alert := exists(is_work) && is_work != true && eventName ==
\"ConsoleLogin\"",
"is_alert := is_alert || (eventName == \"ConsoleLogin\" &&
userIdentity:sessionContext:attributes:mfaAuthenticated == \"False\")",
"is_alert := is_alert || (eventName == \"ConsoleLogin\" &&
additionalEventData:MFAUsed == \"No\")"
]
}
},
"fieldToTypeMap": {},
"config": {},
"triageConfig": {
"riskLevelRules": [
{
"name": "Not WORK",
"comment": "Checks whether the field is_work is true or false.",
"rule": "is_work == false",
"score": 20,
"reason": "FORMAT('%s is not an WORK network!', sourceIPAddress)"
},
{
"name": "MFA",
"comment": "Checks whether MFA used or not.",
"rule": "userIdentity:sessionContext:attributes:mfaAuthenticated ==
'False'",
"score": 20,
"reason": null
},
{
"name": "MFA2",
"comment": "Checks whether MFA used or not.",
"rule": "additionalEventData:MFAUsed == 'No'",
"score": 20,
"reason": null
}
],
"aggregator": "SUM",
"aggregationConfig": {}
}
},
"configuration": {}
}
Any idea why the score isn't 40? I would expect riskLevelRule 1 & 2 to
be SUMmed?
Re: SUM aggregator not working?
Posted by James Sirota <js...@apache.org>.
I think until we officially migrate to ES 5.x you should write code that would be compatible with ES 2.x (if you want that code to be generally consumable by the Metron community).
04.10.2017, 18:04, "Laurens Vets" <la...@daemon.be>:
> It's working now, so I'm happy :)
>
> On 2017-10-04 14:03, Casey Stella wrote:
>> Ok, so this is subtle. Your rules are wrong and I totally understand
>> why
>> you thought they were right.
>>
>> When we index into ES, we take . and convert them to :, however PRIOR
>> to
>> indexing (when threat triage is running) those fields have .'s not :'s
>> Therefore, your rules should be:
>>
>> userIdentity.sessionContext.attributes.mfaAuthenticated == 'False'
>> and
>> additionalEventData.MFAUsed == 'No'
>>
>> The same general argument goes for your threat triage stellar
>> expressions.
>>
>> Sorry about the confusion, we do that mapping because ES doesn't handle
>> those .'s well. Hey, maybe ES 5 is more sane about that sort of thing
>> and
>> we can avoid doing that transformation.
>>
>> Casey
>>
>> On Wed, Oct 4, 2017 at 4:38 PM, Laurens Vets <la...@daemon.be> wrote:
>>
>>> No idea whether it's a bug yet, I just need a 2nd set of eyes :)
>>>
>>> This is my event as indexed in ES (Obviously some parts have been
>>> obfuscated):
>>>
>>> {
>>> "_index": "cloudtrail_index_2017.10.04.19",
>>> "_type": "cloudtrail_doc",
>>> "_id": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
>>> "_score": null,
>>> "_timestamp": 1507143907108,
>>> "_source": {
>>> "eventID": "9e3d5468-2d97-4b9a-9821-5c61fec8c158",
>>> "additionalEventData:MFAUsed": "No",
>>> "adapter:stellaradapter:end:ts": "1507143907145",
>>> "threatinteljoinbolt:joiner:ts": "1507143907153",
>>> "eventVersion": "1.05",
>>> "threat:triage:rules:0:comment": "Checks whether the field is_work
>>> is
>>> true or false.",
>>> "sourceIPAddress": "208.110.73.106",
>>> "eventSource": "signin.amazonaws.com",
>>> "enrichmentsplitterbolt:splitter:begin:ts": "1507143907143",
>>> "enrichmentjoinbolt:joiner:ts": "1507143907147",
>>> "additionalEventData:MobileVersion": "No",
>>> "threat:triage:rules:0:name": "Not WORK",
>>> "source:type": "cloudtrail",
>>> "original_string": "{\"eventVersion\":\"1.05\",\"
>>> userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AIDAI
>>> 5ITCMVR3BQV5DUFW\",\"arn\":\"arn:aws:iam::<ACCOUNTID>:user/
>>> <EMAIL>\",\"accountId\":\"<ACCOUNTID>\",\"userName\":\"<
>>> EMAIL>\"},\"eventTime\":\"2017-10-04T18:57:31Z\",\"eventSource\":\"
>>> signin.amazonaws.com\",\"eventName\":\"ConsoleLogin\"
>>> ,\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"208.110.7
>>> 3.106\",\"userAgent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
>>> rv:56.0)
>>> Gecko/20100101 Firefox/56.0\",\"requestParame
>>> ters\":null,\"responseElements\":{\"ConsoleLogin\":\"
>>> Success\"},\"additionalEventData\":{\"LoginTo\":\"https://
>>> console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true\
>>> <https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true%5C>
>>> ",\"MobileVersion\":\"No\",\"MFAUsed\":\"No\"},\"eventID\":
>>> \"9e3d5468-2d97-4b9a-9821-5c61fec8c158\",\"eventType\":\
>>> "AwsConsoleSignIn\",\"recipientAccountId\":\"<ACCOUNTID>\"}",
>>> "eventTime": "2017-10-04T18:57:31Z",
>>> "eventName": "ConsoleLogin",
>>> "recipientAccountId": "<ACCOUNTID>",
>>> "userIdentity:principalId": "AIDAI5ITCMVR3BQV5DUFW",
>>> "threatintelsplitterbolt:splitter:end:ts": "1507143907148",
>>> "threat:triage:rules:0:score": 20,
>>> "timestamp": 1507143907108,
>>> "threat:triage:rules:0:reason": "208.110.73.106 is not an WORK
>>> network!",
>>> "awsRegion": "us-east-1",
>>> "is_work": false,
>>> "userIdentity:userName": "<EMAIL>",
>>> "enrichmentsplitterbolt:splitter:end:ts": "1507143907143",
>>> "threat:triage:score": 20,
>>> "is_alert": "true",
>>> "userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
>>> Gecko/20100101 Firefox/56.0",
>>> "adapter:stellaradapter:begin:ts": "1507143907145",
>>> "eventType": "AwsConsoleSignIn",
>>> "userIdentity:arn": "arn:aws:iam::<ACCOUNTID>:user/<EMAIL>",
>>> "userIdentity:accountId": "<ACCOUNTID>",
>>> "userIdentity:type": "IAMUser",
>>> "threatintelsplitterbolt:splitter:begin:ts": "1507143907148",
>>> "guid": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
>>> "additionalEventData:LoginTo": "https://console.aws.amazon.co
>>> m/console/home?state=hashArgs%23&isauthcode=true",
>>> "responseElements:ConsoleLogin": "Success"
>>> },
>>> "fields": {
>>> "adapter:stellaradapter:end:ts": [
>>> 1507143907145
>>> ],
>>> "threatinteljoinbolt:joiner:ts": [
>>> 1507143907153
>>> ],
>>> "enrichmentsplitterbolt:splitter:end:ts": [
>>> 1507143907143
>>> ],
>>> "enrichmentsplitterbolt:splitter:begin:ts": [
>>> 1507143907143
>>> ],
>>> "enrichmentjoinbolt:joiner:ts": [
>>> 1507143907147
>>> ],
>>> "adapter:stellaradapter:begin:ts": [
>>> 1507143907145
>>> ],
>>> "eventTime": [
>>> 1507143451000
>>> ],
>>> "threatintelsplitterbolt:splitter:begin:ts": [
>>> 1507143907148
>>> ],
>>> "threatintelsplitterbolt:splitter:end:ts": [
>>> 1507143907148
>>> ],
>>> "timestamp": [
>>> 1507143907108
>>> ]
>>> },
>>> "sort": [
>>> 1507143451000
>>> ]
>>> }
>>>
>>> This is my sensor configuration:
>>>
>>> {
>>> "enrichment": {
>>> "fieldMap": {
>>> "stellar": {
>>> "config": {
>>> "is_work": "IN_SUBNET(if
>>> IS_IP(sourceIPAddress) then sourceIPAddress else NULL, '1.2.3.4/16', '
>>> 5.6.7.8/23')"
>>> }
>>> }
>>> },
>>> "fieldToTypeMap": {},
>>> "config": {}
>>> },
>>> "threatIntel": {
>>> "fieldMap": {
>>> "stellar": {
>>> "config": [
>>> "is_alert := exists(is_work)
>>> &&
>>> is_work != true && eventName == \"ConsoleLogin\"",
>>> "is_alert := is_alert ||
>>> (eventName == \"ConsoleLogin\" &&
>>> userIdentity:sessionContext:attributes:mfaAuthenticated
>>> == \"False\")",
>>> "is_alert := is_alert ||
>>> (eventName == \"ConsoleLogin\" && additionalEventData:MFAUsed ==
>>> \"No\")"
>>> ]
>>> }
>>> },
>>> "fieldToTypeMap": {},
>>> "config": {},
>>> "triageConfig": {
>>> "riskLevelRules": [
>>> {
>>> "name": "Not WORK",
>>> "comment": "Checks whether the
>>> field is_work is true or false.",
>>> "rule": "is_work == false",
>>> "score": 20,
>>> "reason": "FORMAT('%s is not
>>> an
>>> WORK network!', sourceIPAddress)"
>>> },
>>> {
>>> "name": "MFA",
>>> "comment": "Checks whether MFA
>>> used or not.",
>>> "rule":
>>> "userIdentity:sessionContext:attributes:mfaAuthenticated == 'False'",
>>> "score": 20,
>>> "reason": null
>>> },
>>> {
>>> "name": "MFA2",
>>> "comment": "Checks whether MFA
>>> used or not.",
>>> "rule":
>>> "additionalEventData:MFAUsed == 'No'",
>>> "score": 20,
>>> "reason": null
>>> }
>>> ],
>>> "aggregator": "SUM",
>>> "aggregationConfig": {}
>>> }
>>> },
>>> "configuration": {}
>>> }
>>>
>>> Any idea why the score isn't 40? I would expect riskLevelRule 1 & 2 to
>>> be
>>> SUMmed?
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: SUM aggregator not working?
Posted by Laurens Vets <la...@daemon.be>.
It's working now, so I'm happy :)
On 2017-10-04 14:03, Casey Stella wrote:
> Ok, so this is subtle. Your rules are wrong and I totally understand
> why
> you thought they were right.
>
> When we index into ES, we take . and convert them to :, however PRIOR
> to
> indexing (when threat triage is running) those fields have .'s not :'s
> Therefore, your rules should be:
>
> userIdentity.sessionContext.attributes.mfaAuthenticated == 'False'
> and
> additionalEventData.MFAUsed == 'No'
>
> The same general argument goes for your threat triage stellar
> expressions.
>
>
> Sorry about the confusion, we do that mapping because ES doesn't handle
> those .'s well. Hey, maybe ES 5 is more sane about that sort of thing
> and
> we can avoid doing that transformation.
>
> Casey
>
> On Wed, Oct 4, 2017 at 4:38 PM, Laurens Vets <la...@daemon.be> wrote:
>
>> No idea whether it's a bug yet, I just need a 2nd set of eyes :)
>>
>> This is my event as indexed in ES (Obviously some parts have been
>> obfuscated):
>>
>> {
>> "_index": "cloudtrail_index_2017.10.04.19",
>> "_type": "cloudtrail_doc",
>> "_id": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
>> "_score": null,
>> "_timestamp": 1507143907108,
>> "_source": {
>> "eventID": "9e3d5468-2d97-4b9a-9821-5c61fec8c158",
>> "additionalEventData:MFAUsed": "No",
>> "adapter:stellaradapter:end:ts": "1507143907145",
>> "threatinteljoinbolt:joiner:ts": "1507143907153",
>> "eventVersion": "1.05",
>> "threat:triage:rules:0:comment": "Checks whether the field is_work
>> is
>> true or false.",
>> "sourceIPAddress": "208.110.73.106",
>> "eventSource": "signin.amazonaws.com",
>> "enrichmentsplitterbolt:splitter:begin:ts": "1507143907143",
>> "enrichmentjoinbolt:joiner:ts": "1507143907147",
>> "additionalEventData:MobileVersion": "No",
>> "threat:triage:rules:0:name": "Not WORK",
>> "source:type": "cloudtrail",
>> "original_string": "{\"eventVersion\":\"1.05\",\"
>> userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AIDAI
>> 5ITCMVR3BQV5DUFW\",\"arn\":\"arn:aws:iam::<ACCOUNTID>:user/
>> <EMAIL>\",\"accountId\":\"<ACCOUNTID>\",\"userName\":\"<
>> EMAIL>\"},\"eventTime\":\"2017-10-04T18:57:31Z\",\"eventSource\":\"
>> signin.amazonaws.com\",\"eventName\":\"ConsoleLogin\"
>> ,\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"208.110.7
>> 3.106\",\"userAgent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
>> rv:56.0)
>> Gecko/20100101 Firefox/56.0\",\"requestParame
>> ters\":null,\"responseElements\":{\"ConsoleLogin\":\"
>> Success\"},\"additionalEventData\":{\"LoginTo\":\"https://
>> console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true\
>> <https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true%5C>
>> ",\"MobileVersion\":\"No\",\"MFAUsed\":\"No\"},\"eventID\":
>> \"9e3d5468-2d97-4b9a-9821-5c61fec8c158\",\"eventType\":\
>> "AwsConsoleSignIn\",\"recipientAccountId\":\"<ACCOUNTID>\"}",
>> "eventTime": "2017-10-04T18:57:31Z",
>> "eventName": "ConsoleLogin",
>> "recipientAccountId": "<ACCOUNTID>",
>> "userIdentity:principalId": "AIDAI5ITCMVR3BQV5DUFW",
>> "threatintelsplitterbolt:splitter:end:ts": "1507143907148",
>> "threat:triage:rules:0:score": 20,
>> "timestamp": 1507143907108,
>> "threat:triage:rules:0:reason": "208.110.73.106 is not an WORK
>> network!",
>> "awsRegion": "us-east-1",
>> "is_work": false,
>> "userIdentity:userName": "<EMAIL>",
>> "enrichmentsplitterbolt:splitter:end:ts": "1507143907143",
>> "threat:triage:score": 20,
>> "is_alert": "true",
>> "userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
>> Gecko/20100101 Firefox/56.0",
>> "adapter:stellaradapter:begin:ts": "1507143907145",
>> "eventType": "AwsConsoleSignIn",
>> "userIdentity:arn": "arn:aws:iam::<ACCOUNTID>:user/<EMAIL>",
>> "userIdentity:accountId": "<ACCOUNTID>",
>> "userIdentity:type": "IAMUser",
>> "threatintelsplitterbolt:splitter:begin:ts": "1507143907148",
>> "guid": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
>> "additionalEventData:LoginTo": "https://console.aws.amazon.co
>> m/console/home?state=hashArgs%23&isauthcode=true",
>> "responseElements:ConsoleLogin": "Success"
>> },
>> "fields": {
>> "adapter:stellaradapter:end:ts": [
>> 1507143907145
>> ],
>> "threatinteljoinbolt:joiner:ts": [
>> 1507143907153
>> ],
>> "enrichmentsplitterbolt:splitter:end:ts": [
>> 1507143907143
>> ],
>> "enrichmentsplitterbolt:splitter:begin:ts": [
>> 1507143907143
>> ],
>> "enrichmentjoinbolt:joiner:ts": [
>> 1507143907147
>> ],
>> "adapter:stellaradapter:begin:ts": [
>> 1507143907145
>> ],
>> "eventTime": [
>> 1507143451000
>> ],
>> "threatintelsplitterbolt:splitter:begin:ts": [
>> 1507143907148
>> ],
>> "threatintelsplitterbolt:splitter:end:ts": [
>> 1507143907148
>> ],
>> "timestamp": [
>> 1507143907108
>> ]
>> },
>> "sort": [
>> 1507143451000
>> ]
>> }
>>
>> This is my sensor configuration:
>>
>>
>> {
>> "enrichment": {
>> "fieldMap": {
>> "stellar": {
>> "config": {
>> "is_work": "IN_SUBNET(if
>> IS_IP(sourceIPAddress) then sourceIPAddress else NULL, '1.2.3.4/16', '
>> 5.6.7.8/23')"
>> }
>> }
>> },
>> "fieldToTypeMap": {},
>> "config": {}
>> },
>> "threatIntel": {
>> "fieldMap": {
>> "stellar": {
>> "config": [
>> "is_alert := exists(is_work)
>> &&
>> is_work != true && eventName == \"ConsoleLogin\"",
>> "is_alert := is_alert ||
>> (eventName == \"ConsoleLogin\" &&
>> userIdentity:sessionContext:attributes:mfaAuthenticated
>> == \"False\")",
>> "is_alert := is_alert ||
>> (eventName == \"ConsoleLogin\" && additionalEventData:MFAUsed ==
>> \"No\")"
>> ]
>> }
>> },
>> "fieldToTypeMap": {},
>> "config": {},
>> "triageConfig": {
>> "riskLevelRules": [
>> {
>> "name": "Not WORK",
>> "comment": "Checks whether the
>> field is_work is true or false.",
>> "rule": "is_work == false",
>> "score": 20,
>> "reason": "FORMAT('%s is not
>> an
>> WORK network!', sourceIPAddress)"
>> },
>> {
>> "name": "MFA",
>> "comment": "Checks whether MFA
>> used or not.",
>> "rule":
>> "userIdentity:sessionContext:attributes:mfaAuthenticated == 'False'",
>> "score": 20,
>> "reason": null
>> },
>> {
>> "name": "MFA2",
>> "comment": "Checks whether MFA
>> used or not.",
>> "rule":
>> "additionalEventData:MFAUsed == 'No'",
>> "score": 20,
>> "reason": null
>> }
>> ],
>> "aggregator": "SUM",
>> "aggregationConfig": {}
>> }
>> },
>> "configuration": {}
>> }
>>
>> Any idea why the score isn't 40? I would expect riskLevelRule 1 & 2 to
>> be
>> SUMmed?
>>
Re: SUM aggregator not working?
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
You're right, with ES 5 we can use periods directly instead of transforming
them in indexing to colons (actually, this feature was reintroduced sin 2.4
<https://github.com/elastic/elasticsearch/pull/19937/files>). I outlined
this as a benefit in the original JIRA
<https://issues.apache.org/jira/browse/METRON-939?filter=-1>, along with a
ton of other benefits including native IPv6 support </shameless plug>
Jon
On Wed, Oct 4, 2017 at 5:03 PM Casey Stella <ce...@gmail.com> wrote:
> Ok, so this is subtle. Your rules are wrong and I totally understand why
> you thought they were right.
>
> When we index into ES, we take . and convert them to :, however PRIOR to
> indexing (when threat triage is running) those fields have .'s not :'s
> Therefore, your rules should be:
>
> userIdentity.sessionContext.attributes.mfaAuthenticated == 'False'
> and
> additionalEventData.MFAUsed == 'No'
>
> The same general argument goes for your threat triage stellar expressions.
>
>
> Sorry about the confusion, we do that mapping because ES doesn't handle
> those .'s well. Hey, maybe ES 5 is more sane about that sort of thing and
> we can avoid doing that transformation.
>
> Casey
>
> On Wed, Oct 4, 2017 at 4:38 PM, Laurens Vets <la...@daemon.be> wrote:
>
> > No idea whether it's a bug yet, I just need a 2nd set of eyes :)
> >
> > This is my event as indexed in ES (Obviously some parts have been
> > obfuscated):
> >
> > {
> > "_index": "cloudtrail_index_2017.10.04.19",
> > "_type": "cloudtrail_doc",
> > "_id": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
> > "_score": null,
> > "_timestamp": 1507143907108,
> > "_source": {
> > "eventID": "9e3d5468-2d97-4b9a-9821-5c61fec8c158",
> > "additionalEventData:MFAUsed": "No",
> > "adapter:stellaradapter:end:ts": "1507143907145",
> > "threatinteljoinbolt:joiner:ts": "1507143907153",
> > "eventVersion": "1.05",
> > "threat:triage:rules:0:comment": "Checks whether the field is_work is
> > true or false.",
> > "sourceIPAddress": "208.110.73.106",
> > "eventSource": "signin.amazonaws.com",
> > "enrichmentsplitterbolt:splitter:begin:ts": "1507143907143",
> > "enrichmentjoinbolt:joiner:ts": "1507143907147",
> > "additionalEventData:MobileVersion": "No",
> > "threat:triage:rules:0:name": "Not WORK",
> > "source:type": "cloudtrail",
> > "original_string": "{\"eventVersion\":\"1.05\",\"
> > userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AIDAI
> > 5ITCMVR3BQV5DUFW\",\"arn\":\"arn:aws:iam::<ACCOUNTID>:user/
> > <EMAIL>\",\"accountId\":\"<ACCOUNTID>\",\"userName\":\"<
> > EMAIL>\"},\"eventTime\":\"2017-10-04T18:57:31Z\",\"eventSource\":\"
> > signin.amazonaws.com\",\"eventName\":\"ConsoleLogin\"
> > ,\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"208.110.7
> > 3.106\",\"userAgent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
> > Gecko/20100101 Firefox/56.0\",\"requestParame
> > ters\":null,\"responseElements\":{\"ConsoleLogin\":\"
> > Success\"},\"additionalEventData\":{\"LoginTo\":\"https://
> > console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true\
> <http://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true%5C>
> > <
> https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true%5C
> >
> > ",\"MobileVersion\":\"No\",\"MFAUsed\":\"No\"},\"eventID\":
> > \"9e3d5468-2d97-4b9a-9821-5c61fec8c158\",\"eventType\":\
> > "AwsConsoleSignIn\",\"recipientAccountId\":\"<ACCOUNTID>\"}",
> > "eventTime": "2017-10-04T18:57:31Z",
> > "eventName": "ConsoleLogin",
> > "recipientAccountId": "<ACCOUNTID>",
> > "userIdentity:principalId": "AIDAI5ITCMVR3BQV5DUFW",
> > "threatintelsplitterbolt:splitter:end:ts": "1507143907148",
> > "threat:triage:rules:0:score": 20,
> > "timestamp": 1507143907108,
> > "threat:triage:rules:0:reason": "208.110.73.106 is not an WORK
> > network!",
> > "awsRegion": "us-east-1",
> > "is_work": false,
> > "userIdentity:userName": "<EMAIL>",
> > "enrichmentsplitterbolt:splitter:end:ts": "1507143907143",
> > "threat:triage:score": 20,
> > "is_alert": "true",
> > "userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
> > Gecko/20100101 Firefox/56.0",
> > "adapter:stellaradapter:begin:ts": "1507143907145",
> > "eventType": "AwsConsoleSignIn",
> > "userIdentity:arn": "arn:aws:iam::<ACCOUNTID>:user/<EMAIL>",
> > "userIdentity:accountId": "<ACCOUNTID>",
> > "userIdentity:type": "IAMUser",
> > "threatintelsplitterbolt:splitter:begin:ts": "1507143907148",
> > "guid": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
> > "additionalEventData:LoginTo": "https://console.aws.amazon.co
> > m/console/home?state=hashArgs%23&isauthcode=true",
> > "responseElements:ConsoleLogin": "Success"
> > },
> > "fields": {
> > "adapter:stellaradapter:end:ts": [
> > 1507143907145
> > ],
> > "threatinteljoinbolt:joiner:ts": [
> > 1507143907153
> > ],
> > "enrichmentsplitterbolt:splitter:end:ts": [
> > 1507143907143
> > ],
> > "enrichmentsplitterbolt:splitter:begin:ts": [
> > 1507143907143
> > ],
> > "enrichmentjoinbolt:joiner:ts": [
> > 1507143907147
> > ],
> > "adapter:stellaradapter:begin:ts": [
> > 1507143907145
> > ],
> > "eventTime": [
> > 1507143451000
> > ],
> > "threatintelsplitterbolt:splitter:begin:ts": [
> > 1507143907148
> > ],
> > "threatintelsplitterbolt:splitter:end:ts": [
> > 1507143907148
> > ],
> > "timestamp": [
> > 1507143907108
> > ]
> > },
> > "sort": [
> > 1507143451000
> > ]
> > }
> >
> > This is my sensor configuration:
> >
> >
> > {
> > "enrichment": {
> > "fieldMap": {
> > "stellar": {
> > "config": {
> > "is_work": "IN_SUBNET(if
> > IS_IP(sourceIPAddress) then sourceIPAddress else NULL, '1.2.3.4/16', '
> > 5.6.7.8/23')"
> > }
> > }
> > },
> > "fieldToTypeMap": {},
> > "config": {}
> > },
> > "threatIntel": {
> > "fieldMap": {
> > "stellar": {
> > "config": [
> > "is_alert := exists(is_work) &&
> > is_work != true && eventName == \"ConsoleLogin\"",
> > "is_alert := is_alert ||
> > (eventName == \"ConsoleLogin\" &&
> userIdentity:sessionContext:attributes:mfaAuthenticated
> > == \"False\")",
> > "is_alert := is_alert ||
> > (eventName == \"ConsoleLogin\" && additionalEventData:MFAUsed == \"No\")"
> > ]
> > }
> > },
> > "fieldToTypeMap": {},
> > "config": {},
> > "triageConfig": {
> > "riskLevelRules": [
> > {
> > "name": "Not WORK",
> > "comment": "Checks whether the
> > field is_work is true or false.",
> > "rule": "is_work == false",
> > "score": 20,
> > "reason": "FORMAT('%s is not an
> > WORK network!', sourceIPAddress)"
> > },
> > {
> > "name": "MFA",
> > "comment": "Checks whether MFA
> > used or not.",
> > "rule":
> > "userIdentity:sessionContext:attributes:mfaAuthenticated == 'False'",
> > "score": 20,
> > "reason": null
> > },
> > {
> > "name": "MFA2",
> > "comment": "Checks whether MFA
> > used or not.",
> > "rule":
> > "additionalEventData:MFAUsed == 'No'",
> > "score": 20,
> > "reason": null
> > }
> > ],
> > "aggregator": "SUM",
> > "aggregationConfig": {}
> > }
> > },
> > "configuration": {}
> > }
> >
> > Any idea why the score isn't 40? I would expect riskLevelRule 1 & 2 to be
> > SUMmed?
> >
>
--
Jon
Re: SUM aggregator not working?
Posted by Casey Stella <ce...@gmail.com>.
Ok, so this is subtle. Your rules are wrong and I totally understand why
you thought they were right.
When we index into ES, we take . and convert them to :, however PRIOR to
indexing (when threat triage is running) those fields have .'s not :'s
Therefore, your rules should be:
userIdentity.sessionContext.attributes.mfaAuthenticated == 'False'
and
additionalEventData.MFAUsed == 'No'
The same general argument goes for your threat triage stellar expressions.
Sorry about the confusion, we do that mapping because ES doesn't handle
those .'s well. Hey, maybe ES 5 is more sane about that sort of thing and
we can avoid doing that transformation.
Casey
On Wed, Oct 4, 2017 at 4:38 PM, Laurens Vets <la...@daemon.be> wrote:
> No idea whether it's a bug yet, I just need a 2nd set of eyes :)
>
> This is my event as indexed in ES (Obviously some parts have been
> obfuscated):
>
> {
> "_index": "cloudtrail_index_2017.10.04.19",
> "_type": "cloudtrail_doc",
> "_id": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
> "_score": null,
> "_timestamp": 1507143907108,
> "_source": {
> "eventID": "9e3d5468-2d97-4b9a-9821-5c61fec8c158",
> "additionalEventData:MFAUsed": "No",
> "adapter:stellaradapter:end:ts": "1507143907145",
> "threatinteljoinbolt:joiner:ts": "1507143907153",
> "eventVersion": "1.05",
> "threat:triage:rules:0:comment": "Checks whether the field is_work is
> true or false.",
> "sourceIPAddress": "208.110.73.106",
> "eventSource": "signin.amazonaws.com",
> "enrichmentsplitterbolt:splitter:begin:ts": "1507143907143",
> "enrichmentjoinbolt:joiner:ts": "1507143907147",
> "additionalEventData:MobileVersion": "No",
> "threat:triage:rules:0:name": "Not WORK",
> "source:type": "cloudtrail",
> "original_string": "{\"eventVersion\":\"1.05\",\"
> userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AIDAI
> 5ITCMVR3BQV5DUFW\",\"arn\":\"arn:aws:iam::<ACCOUNTID>:user/
> <EMAIL>\",\"accountId\":\"<ACCOUNTID>\",\"userName\":\"<
> EMAIL>\"},\"eventTime\":\"2017-10-04T18:57:31Z\",\"eventSource\":\"
> signin.amazonaws.com\",\"eventName\":\"ConsoleLogin\"
> ,\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"208.110.7
> 3.106\",\"userAgent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
> Gecko/20100101 Firefox/56.0\",\"requestParame
> ters\":null,\"responseElements\":{\"ConsoleLogin\":\"
> Success\"},\"additionalEventData\":{\"LoginTo\":\"https://
> console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true\
> <https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true%5C>
> ",\"MobileVersion\":\"No\",\"MFAUsed\":\"No\"},\"eventID\":
> \"9e3d5468-2d97-4b9a-9821-5c61fec8c158\",\"eventType\":\
> "AwsConsoleSignIn\",\"recipientAccountId\":\"<ACCOUNTID>\"}",
> "eventTime": "2017-10-04T18:57:31Z",
> "eventName": "ConsoleLogin",
> "recipientAccountId": "<ACCOUNTID>",
> "userIdentity:principalId": "AIDAI5ITCMVR3BQV5DUFW",
> "threatintelsplitterbolt:splitter:end:ts": "1507143907148",
> "threat:triage:rules:0:score": 20,
> "timestamp": 1507143907108,
> "threat:triage:rules:0:reason": "208.110.73.106 is not an WORK
> network!",
> "awsRegion": "us-east-1",
> "is_work": false,
> "userIdentity:userName": "<EMAIL>",
> "enrichmentsplitterbolt:splitter:end:ts": "1507143907143",
> "threat:triage:score": 20,
> "is_alert": "true",
> "userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
> Gecko/20100101 Firefox/56.0",
> "adapter:stellaradapter:begin:ts": "1507143907145",
> "eventType": "AwsConsoleSignIn",
> "userIdentity:arn": "arn:aws:iam::<ACCOUNTID>:user/<EMAIL>",
> "userIdentity:accountId": "<ACCOUNTID>",
> "userIdentity:type": "IAMUser",
> "threatintelsplitterbolt:splitter:begin:ts": "1507143907148",
> "guid": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
> "additionalEventData:LoginTo": "https://console.aws.amazon.co
> m/console/home?state=hashArgs%23&isauthcode=true",
> "responseElements:ConsoleLogin": "Success"
> },
> "fields": {
> "adapter:stellaradapter:end:ts": [
> 1507143907145
> ],
> "threatinteljoinbolt:joiner:ts": [
> 1507143907153
> ],
> "enrichmentsplitterbolt:splitter:end:ts": [
> 1507143907143
> ],
> "enrichmentsplitterbolt:splitter:begin:ts": [
> 1507143907143
> ],
> "enrichmentjoinbolt:joiner:ts": [
> 1507143907147
> ],
> "adapter:stellaradapter:begin:ts": [
> 1507143907145
> ],
> "eventTime": [
> 1507143451000
> ],
> "threatintelsplitterbolt:splitter:begin:ts": [
> 1507143907148
> ],
> "threatintelsplitterbolt:splitter:end:ts": [
> 1507143907148
> ],
> "timestamp": [
> 1507143907108
> ]
> },
> "sort": [
> 1507143451000
> ]
> }
>
> This is my sensor configuration:
>
>
> {
> "enrichment": {
> "fieldMap": {
> "stellar": {
> "config": {
> "is_work": "IN_SUBNET(if
> IS_IP(sourceIPAddress) then sourceIPAddress else NULL, '1.2.3.4/16', '
> 5.6.7.8/23')"
> }
> }
> },
> "fieldToTypeMap": {},
> "config": {}
> },
> "threatIntel": {
> "fieldMap": {
> "stellar": {
> "config": [
> "is_alert := exists(is_work) &&
> is_work != true && eventName == \"ConsoleLogin\"",
> "is_alert := is_alert ||
> (eventName == \"ConsoleLogin\" && userIdentity:sessionContext:attributes:mfaAuthenticated
> == \"False\")",
> "is_alert := is_alert ||
> (eventName == \"ConsoleLogin\" && additionalEventData:MFAUsed == \"No\")"
> ]
> }
> },
> "fieldToTypeMap": {},
> "config": {},
> "triageConfig": {
> "riskLevelRules": [
> {
> "name": "Not WORK",
> "comment": "Checks whether the
> field is_work is true or false.",
> "rule": "is_work == false",
> "score": 20,
> "reason": "FORMAT('%s is not an
> WORK network!', sourceIPAddress)"
> },
> {
> "name": "MFA",
> "comment": "Checks whether MFA
> used or not.",
> "rule":
> "userIdentity:sessionContext:attributes:mfaAuthenticated == 'False'",
> "score": 20,
> "reason": null
> },
> {
> "name": "MFA2",
> "comment": "Checks whether MFA
> used or not.",
> "rule":
> "additionalEventData:MFAUsed == 'No'",
> "score": 20,
> "reason": null
> }
> ],
> "aggregator": "SUM",
> "aggregationConfig": {}
> }
> },
> "configuration": {}
> }
>
> Any idea why the score isn't 40? I would expect riskLevelRule 1 & 2 to be
> SUMmed?
>