You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Nick Allen (JIRA)" <ji...@apache.org> on 2018/04/20 20:14:00 UTC
[jira] [Updated] (METRON-1533) Create KAFKA_FIND Stellar Function
[ https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Allen updated METRON-1533:
-------------------------------
Description:
When creating enrichments, I often find that I want to validate that the enrichment I just created was successful on the live, incoming stream of telemetry. My workflow looks something like this.
1. Create and test the enrichment that I want to create.
{code:java}
[Stellar]>>> ip_src_addr := "72.34.49.86"
72.34.49.86
[Stellar]>>> geo := GEO_GET(ip_src_addr)
{country=US, dmaCode=803, city=Los Angeles, postalCode=90014, latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, longitude=-118.2512}
{code}
2. That looks good to me. Now let's add that to my Bro telemetry.
{code:java}
[Stellar]>>> conf := SHELL_EDIT(conf)
{
"enrichment" : {
"fieldMap": {
"stellar": {
"config": [
"geo := GEO_GET(ip_src_addr)"
]
}
}
},
"threatIntel": {
}
}
[Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
{code}
3. It looks like that worked, but did that really work?
At this point, I would run KAFKA_GET as many times as it takes to retrieve a Bro message. You would just have to get lucky and hope that the enrichment worked and secondly that you would pull down a Bro message (as opposed to a different sensor).
I would rather have a function that lets me only pull back the messages that I care about. In this case I could either retrieve only Bro messages.
{code:java}
KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
{code}
Or I could look for messages that contain geolocation data.
{code:java}
KAFKA_FIND('indexing', m -> MAP_EXISTS('geo', m))
{code}
was:
When creating enrichments, I often find that I want to validate that the enrichment I just created was successful on the live, incoming stream of telemetry. My workflow looks something like this.
1. Create and test the enrichment that I want to create.
{code:java}
[Stellar]>>> ip_src_addr := "72.34.49.86"
72.34.49.86
[Stellar]>>> geo := GEO_GET(ip_src_addr)
{country=US, dmaCode=803, city=Los Angeles, postalCode=90014, latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, longitude=-118.2512}
{code}
2. That looks good to me. Now let's add that to my Bro telemetry.
{code:java}
[Stellar]>>> conf := SHELL_EDIT(conf)
{
"enrichment" : {
"fieldMap": {
"stellar": {
"config": [
"geo := GEO_GET(ip_src_addr)"
]
}
}
},
"threatIntel": {
}
}
[Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
{code}
3. It looks like that worked, but did that really work?
At this point, I would run KAFKA_GET as many times as it takes to retrieve a Bro message. You would just have to get lucky and hope that the enrichment worked and secondly that you would pull down a Bro message (as opposed to a different sensor).
4. I would rather have a function that lets me only pull back the messages that I care about. In this case I could either retrieve only Bro messages.
{code:java}
KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
{code}
Or I could look for messages that contain geolocation data.
{code:java}
KAFKA_FIND('topic', m -> MAP_EXISTS('geo', m))
{code}
> Create KAFKA_FIND Stellar Function
> ----------------------------------
>
> Key: METRON-1533
> URL: https://issues.apache.org/jira/browse/METRON-1533
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Minor
>
> When creating enrichments, I often find that I want to validate that the enrichment I just created was successful on the live, incoming stream of telemetry. My workflow looks something like this.
> 1. Create and test the enrichment that I want to create.
> {code:java}
> [Stellar]>>> ip_src_addr := "72.34.49.86"
> 72.34.49.86
> [Stellar]>>> geo := GEO_GET(ip_src_addr)
> {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, longitude=-118.2512}
> {code}
> 2. That looks good to me. Now let's add that to my Bro telemetry.
> {code:java}
> [Stellar]>>> conf := SHELL_EDIT(conf)
> {
> "enrichment" : {
> "fieldMap": {
> "stellar": {
> "config": [
> "geo := GEO_GET(ip_src_addr)"
> ]
> }
> }
> },
> "threatIntel": {
> }
> }
> [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
> {code}
>
> 3. It looks like that worked, but did that really work?
> At this point, I would run KAFKA_GET as many times as it takes to retrieve a Bro message. You would just have to get lucky and hope that the enrichment worked and secondly that you would pull down a Bro message (as opposed to a different sensor).
>
> I would rather have a function that lets me only pull back the messages that I care about. In this case I could either retrieve only Bro messages.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
> {code}
> Or I could look for messages that contain geolocation data.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_EXISTS('geo', m))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)