You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Geeks Girls <ge...@gmail.com> on 2019/12/04 19:35:58 UTC
Data not populating in metron alerts ui
Hi,
I am planning to use Metron as a SIEM and exploring it's features. Thanks
for the great documentation. It helped a lot to set it up quickly.
Initially configured snort ,bro,yaf logs to flow into Metron . For snort,
could see threat triage rules configured in the Metron enrichment config.
But for bro logs, is_alert field is blank .I verified the data in the
Kibana. Though the is_alert is blank ,those logs also appearing in metron
alerts ui.How this could be possible.
Also i pushed json data to the new datasource which i configured . I
created a new elastic search template. When reading the readme file, it has
been mentioned to create metron_alert field. What is the difference between
is_alert and metron_alert.
What are the configuration needed to push data as alerts in metron
AlertsUI. I could see logs are being parsed,enriched and indexed in the
elastic search. So created Kibana dashboard .But i couldn't find anything
i. AlertsUI. What should i do?Any help is highly appreciated.
Thanks,
Jai
Re: Data not populating in metron alerts ui
Posted by Geeks Girls <ge...@gmail.com>.
Hi Nick,
Thanks for the explanation. I want to integrate some threat feeds and also
trying out MaaS . I saw like those are done during enrichment.If we can
pass a batch of data to the threat feed endpoints or MaaS endpoint, that
would be great.We have a custom threat feed, i need to pass a telemetry
source data to that custom feed for enrichment.It can accept 100k records
for processing. So i thought if i can pass set of records to that ,it would
help. Also for MaaS, I can deploy some machine learning models. So batch
enrichment should be helpful.
Regards
Jai
On Fri, 6 Dec, 2019, 4:47 AM Nick Allen, <ni...@nickallen.org> wrote:
> Hi Jai -
>
> > Metron is a SIEM tool and by default if i add is_alert as true,then how
> can i really leverage Metron as a SIEM . please correct me if i am
> wrong.All data will be pushed as alerts right.
>
> You should set `is_alert` to a Stellar expression using whatever logic is
> appropriate for your use case. This lets you determine what is important
> enough for you to triage. You wouldn't just set it to true for all
> telemetry, unless you do indeed want to triage everything. All telemetry
> gets indexed for search (Solr/Elasticsearch) and analytics (HDFS).
>
> > How about batch enrichment. Is it possible to use batch during
> enrichment.
>
> No, we do enrichment on the streaming telemetry. Keep in mind many
> enrichments can be time sensitive. What sort of use case do you have in
> mind for this?
>
> Hope this helps
>
>
>
> On Thu, Dec 5, 2019 at 11:48 AM Geeks Girls <ge...@gmail.com>
> wrote:
>
>> Hi Sanket,
>>
>>
>> Thanks a lot for the explanation. I am able to see the logs after adding
>> an additional column is_alert to true. Metron is a SIEM tool and by default
>> if i add is_alert as true,then how can i really leverage Metron as a SIEM .
>> please correct me if i am wrong.All data will be pushed as alerts right.
>>
>> Also i could see configuration for batch indexing in json file. How about
>> batch enrichment. Is it possible to use batch during enrichment.
>>
>> Thanks,
>> Jai
>>
>>
>> On Thu, 5 Dec, 2019, 4:14 AM Sanket Sharma, <sa...@dukstra.com>
>> wrote:
>>
>>> Hi Jai,
>>>
>>>
>>>
>>> Please see my responses below:
>>>
>>>
>>>
>>> >>>>>>>“But for bro logs, is_alert field is blank .I verified the data
>>> in the Kibana. Though the is_alert is blank ,those logs also appearing in
>>> metron alerts ui.How this could be possible.”
>>>
>>>
>>>
>>> This confused me in the beginning as well, but “is_alert” field only
>>> controls threat triage score and does not change the behaviour of what gets
>>> display in the alerts UI. If the is_alert field is set to true, then threat
>>> triage score is calculated based on the rules specified. If is_alert is set
>>> to false, then score calculation is skipped. It has no effect on indexing
>>> and does not change the behaviour of what gets displayed in the alerts UI.
>>> As @Nick Allen <ni...@nickallen.org> explained to me earlier:
>>>
>>>
>>>
>>> “Threat Triage only runs on messages where there is a field named
>>> "is_alert" with a Stellar expression that evaluates to true. This allows
>>> you to avoid the expense of Threat Triage, in cases where you know it is
>>> not needed. The "is_alert" field is used as a flag to indicate which
>>> messages should undergo Threat Triage processing. Once a message completes
>>> Threat Triage and gets a score, changing or removing the "is_alert" field
>>> does not do anything. “
>>>
>>>
>>>
>>>
>>>
>>> >>>>>>>” Also i pushed json data to the new datasource which i
>>> configured . I created a new elastic search template. When reading the
>>> readme file, it has been mentioned to create metron_alert field. What is
>>> the difference between is_alert and metron_alert.”
>>>
>>>
>>>
>>> As explained earlier, the is_alert field is required on the incoming
>>> message/event from the telemetry source. If the field is present and set to
>>> true, then the event/message will be undergo threat triage processing and a
>>> score will be assigned to it using the rules defined. “metron_alert” field
>>> on the other hand is required on the index template on Elasticsearch (not
>>> on the incoming event/message). I stand to be corrected, but there are some
>>> references which mention that this is a dummy field (
>>> https://metron.apache.org/current-book/metron-platform/metron-elasticsearch/index.html
>>> )
>>>
>>>
>>>
>>> >>>>>>>” But i couldn't find anything i. AlertsUI”
>>>
>>>
>>>
>>> Usually it is because of a missing field that is required by
>>> Elasticsearch and metron. You may refer to documentation here
>>> https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch/metron-elasticsearch-common
>>> and
>>> https://docs.cloudera.com/HDPDocuments/HCP1/HCP-1.9.0/add-new-telemetry-data-source/content/create_elasticsearch_index_template.html
>>>
>>> Refer to section “Elastic Search” and “Using Metron with ElasticSearch
>>> 5.6” for details. Please also refer to metron-rest logs. If you still face
>>> issues, please include your index template in your response.
>>>
>>>
>>>
>>> Hope that helps.
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Sanket
>>>
>>>
>>>
>>> *From: *Geeks Girls <ge...@gmail.com>
>>> *Reply to: *"user@metron.apache.org" <us...@metron.apache.org>
>>> *Date: *Wednesday, 04 December 2019 at 21:34
>>> *To: *"user@metron.apache.org" <us...@metron.apache.org>
>>> *Subject: *Data not populating in metron alerts ui
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> I am planning to use Metron as a SIEM and exploring it's features.
>>> Thanks for the great documentation. It helped a lot to set it up quickly.
>>> Initially configured snort ,bro,yaf logs to flow into Metron . For snort,
>>> could see threat triage rules configured in the Metron enrichment config.
>>> But for bro logs, is_alert field is blank .I verified the data in the
>>> Kibana. Though the is_alert is blank ,those logs also appearing in metron
>>> alerts ui.How this could be possible.
>>>
>>> Also i pushed json data to the new datasource which i configured . I
>>> created a new elastic search template. When reading the readme file, it has
>>> been mentioned to create metron_alert field. What is the difference between
>>> is_alert and metron_alert.
>>>
>>>
>>>
>>> What are the configuration needed to push data as alerts in metron
>>> AlertsUI. I could see logs are being parsed,enriched and indexed in the
>>> elastic search. So created Kibana dashboard .But i couldn't find anything
>>> i. AlertsUI. What should i do?Any help is highly appreciated.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Jai
>>>
>>
Re: Data not populating in metron alerts ui
Posted by Nick Allen <ni...@nickallen.org>.
Hi Jai -
> Metron is a SIEM tool and by default if i add is_alert as true,then how
can i really leverage Metron as a SIEM . please correct me if i am
wrong.All data will be pushed as alerts right.
You should set `is_alert` to a Stellar expression using whatever logic is
appropriate for your use case. This lets you determine what is important
enough for you to triage. You wouldn't just set it to true for all
telemetry, unless you do indeed want to triage everything. All telemetry
gets indexed for search (Solr/Elasticsearch) and analytics (HDFS).
> How about batch enrichment. Is it possible to use batch during
enrichment.
No, we do enrichment on the streaming telemetry. Keep in mind many
enrichments can be time sensitive. What sort of use case do you have in
mind for this?
Hope this helps
On Thu, Dec 5, 2019 at 11:48 AM Geeks Girls <ge...@gmail.com> wrote:
> Hi Sanket,
>
>
> Thanks a lot for the explanation. I am able to see the logs after adding
> an additional column is_alert to true. Metron is a SIEM tool and by default
> if i add is_alert as true,then how can i really leverage Metron as a SIEM .
> please correct me if i am wrong.All data will be pushed as alerts right.
>
> Also i could see configuration for batch indexing in json file. How about
> batch enrichment. Is it possible to use batch during enrichment.
>
> Thanks,
> Jai
>
>
> On Thu, 5 Dec, 2019, 4:14 AM Sanket Sharma, <sa...@dukstra.com>
> wrote:
>
>> Hi Jai,
>>
>>
>>
>> Please see my responses below:
>>
>>
>>
>> >>>>>>>“But for bro logs, is_alert field is blank .I verified the data
>> in the Kibana. Though the is_alert is blank ,those logs also appearing in
>> metron alerts ui.How this could be possible.”
>>
>>
>>
>> This confused me in the beginning as well, but “is_alert” field only
>> controls threat triage score and does not change the behaviour of what gets
>> display in the alerts UI. If the is_alert field is set to true, then threat
>> triage score is calculated based on the rules specified. If is_alert is set
>> to false, then score calculation is skipped. It has no effect on indexing
>> and does not change the behaviour of what gets displayed in the alerts UI.
>> As @Nick Allen <ni...@nickallen.org> explained to me earlier:
>>
>>
>>
>> “Threat Triage only runs on messages where there is a field named
>> "is_alert" with a Stellar expression that evaluates to true. This allows
>> you to avoid the expense of Threat Triage, in cases where you know it is
>> not needed. The "is_alert" field is used as a flag to indicate which
>> messages should undergo Threat Triage processing. Once a message completes
>> Threat Triage and gets a score, changing or removing the "is_alert" field
>> does not do anything. “
>>
>>
>>
>>
>>
>> >>>>>>>” Also i pushed json data to the new datasource which i
>> configured . I created a new elastic search template. When reading the
>> readme file, it has been mentioned to create metron_alert field. What is
>> the difference between is_alert and metron_alert.”
>>
>>
>>
>> As explained earlier, the is_alert field is required on the incoming
>> message/event from the telemetry source. If the field is present and set to
>> true, then the event/message will be undergo threat triage processing and a
>> score will be assigned to it using the rules defined. “metron_alert” field
>> on the other hand is required on the index template on Elasticsearch (not
>> on the incoming event/message). I stand to be corrected, but there are some
>> references which mention that this is a dummy field (
>> https://metron.apache.org/current-book/metron-platform/metron-elasticsearch/index.html
>> )
>>
>>
>>
>> >>>>>>>” But i couldn't find anything i. AlertsUI”
>>
>>
>>
>> Usually it is because of a missing field that is required by
>> Elasticsearch and metron. You may refer to documentation here
>> https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch/metron-elasticsearch-common
>> and
>> https://docs.cloudera.com/HDPDocuments/HCP1/HCP-1.9.0/add-new-telemetry-data-source/content/create_elasticsearch_index_template.html
>>
>> Refer to section “Elastic Search” and “Using Metron with ElasticSearch
>> 5.6” for details. Please also refer to metron-rest logs. If you still face
>> issues, please include your index template in your response.
>>
>>
>>
>> Hope that helps.
>>
>>
>>
>> Best regards,
>>
>> Sanket
>>
>>
>>
>> *From: *Geeks Girls <ge...@gmail.com>
>> *Reply to: *"user@metron.apache.org" <us...@metron.apache.org>
>> *Date: *Wednesday, 04 December 2019 at 21:34
>> *To: *"user@metron.apache.org" <us...@metron.apache.org>
>> *Subject: *Data not populating in metron alerts ui
>>
>>
>>
>> Hi,
>>
>>
>>
>> I am planning to use Metron as a SIEM and exploring it's features. Thanks
>> for the great documentation. It helped a lot to set it up quickly.
>> Initially configured snort ,bro,yaf logs to flow into Metron . For snort,
>> could see threat triage rules configured in the Metron enrichment config.
>> But for bro logs, is_alert field is blank .I verified the data in the
>> Kibana. Though the is_alert is blank ,those logs also appearing in metron
>> alerts ui.How this could be possible.
>>
>> Also i pushed json data to the new datasource which i configured . I
>> created a new elastic search template. When reading the readme file, it has
>> been mentioned to create metron_alert field. What is the difference between
>> is_alert and metron_alert.
>>
>>
>>
>> What are the configuration needed to push data as alerts in metron
>> AlertsUI. I could see logs are being parsed,enriched and indexed in the
>> elastic search. So created Kibana dashboard .But i couldn't find anything
>> i. AlertsUI. What should i do?Any help is highly appreciated.
>>
>>
>>
>> Thanks,
>>
>> Jai
>>
>
Re: Data not populating in metron alerts ui
Posted by Geeks Girls <ge...@gmail.com>.
Hi Sanket,
Thanks a lot for the explanation. I am able to see the logs after adding an
additional column is_alert to true. Metron is a SIEM tool and by default if
i add is_alert as true,then how can i really leverage Metron as a SIEM .
please correct me if i am wrong.All data will be pushed as alerts right.
Also i could see configuration for batch indexing in json file. How about
batch enrichment. Is it possible to use batch during enrichment.
Thanks,
Jai
On Thu, 5 Dec, 2019, 4:14 AM Sanket Sharma, <sa...@dukstra.com>
wrote:
> Hi Jai,
>
>
>
> Please see my responses below:
>
>
>
> >>>>>>>“But for bro logs, is_alert field is blank .I verified the data in
> the Kibana. Though the is_alert is blank ,those logs also appearing in
> metron alerts ui.How this could be possible.”
>
>
>
> This confused me in the beginning as well, but “is_alert” field only
> controls threat triage score and does not change the behaviour of what gets
> display in the alerts UI. If the is_alert field is set to true, then threat
> triage score is calculated based on the rules specified. If is_alert is set
> to false, then score calculation is skipped. It has no effect on indexing
> and does not change the behaviour of what gets displayed in the alerts UI.
> As @Nick Allen <ni...@nickallen.org> explained to me earlier:
>
>
>
> “Threat Triage only runs on messages where there is a field named
> "is_alert" with a Stellar expression that evaluates to true. This allows
> you to avoid the expense of Threat Triage, in cases where you know it is
> not needed. The "is_alert" field is used as a flag to indicate which
> messages should undergo Threat Triage processing. Once a message completes
> Threat Triage and gets a score, changing or removing the "is_alert" field
> does not do anything. “
>
>
>
>
>
> >>>>>>>” Also i pushed json data to the new datasource which i configured
> . I created a new elastic search template. When reading the readme file, it
> has been mentioned to create metron_alert field. What is the difference
> between is_alert and metron_alert.”
>
>
>
> As explained earlier, the is_alert field is required on the incoming
> message/event from the telemetry source. If the field is present and set to
> true, then the event/message will be undergo threat triage processing and a
> score will be assigned to it using the rules defined. “metron_alert” field
> on the other hand is required on the index template on Elasticsearch (not
> on the incoming event/message). I stand to be corrected, but there are some
> references which mention that this is a dummy field (
> https://metron.apache.org/current-book/metron-platform/metron-elasticsearch/index.html
> )
>
>
>
> >>>>>>>” But i couldn't find anything i. AlertsUI”
>
>
>
> Usually it is because of a missing field that is required by Elasticsearch
> and metron. You may refer to documentation here
> https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch/metron-elasticsearch-common
> and
> https://docs.cloudera.com/HDPDocuments/HCP1/HCP-1.9.0/add-new-telemetry-data-source/content/create_elasticsearch_index_template.html
>
> Refer to section “Elastic Search” and “Using Metron with ElasticSearch
> 5.6” for details. Please also refer to metron-rest logs. If you still face
> issues, please include your index template in your response.
>
>
>
> Hope that helps.
>
>
>
> Best regards,
>
> Sanket
>
>
>
> *From: *Geeks Girls <ge...@gmail.com>
> *Reply to: *"user@metron.apache.org" <us...@metron.apache.org>
> *Date: *Wednesday, 04 December 2019 at 21:34
> *To: *"user@metron.apache.org" <us...@metron.apache.org>
> *Subject: *Data not populating in metron alerts ui
>
>
>
> Hi,
>
>
>
> I am planning to use Metron as a SIEM and exploring it's features. Thanks
> for the great documentation. It helped a lot to set it up quickly.
> Initially configured snort ,bro,yaf logs to flow into Metron . For snort,
> could see threat triage rules configured in the Metron enrichment config.
> But for bro logs, is_alert field is blank .I verified the data in the
> Kibana. Though the is_alert is blank ,those logs also appearing in metron
> alerts ui.How this could be possible.
>
> Also i pushed json data to the new datasource which i configured . I
> created a new elastic search template. When reading the readme file, it has
> been mentioned to create metron_alert field. What is the difference between
> is_alert and metron_alert.
>
>
>
> What are the configuration needed to push data as alerts in metron
> AlertsUI. I could see logs are being parsed,enriched and indexed in the
> elastic search. So created Kibana dashboard .But i couldn't find anything
> i. AlertsUI. What should i do?Any help is highly appreciated.
>
>
>
> Thanks,
>
> Jai
>
Re: Data not populating in metron alerts ui
Posted by Sanket Sharma <sa...@dukstra.com>.
Hi Jai,
Please see my responses below:
>>>>>>>“But for bro logs, is_alert field is blank .I verified the data in the Kibana. Though the is_alert is blank ,those logs also appearing in metron alerts ui.How this could be possible.”
This confused me in the beginning as well, but “is_alert” field only controls threat triage score and does not change the behaviour of what gets display in the alerts UI. If the is_alert field is set to true, then threat triage score is calculated based on the rules specified. If is_alert is set to false, then score calculation is skipped. It has no effect on indexing and does not change the behaviour of what gets displayed in the alerts UI. As @Nick Allen<ma...@nickallen.org> explained to me earlier:
“Threat Triage only runs on messages where there is a field named "is_alert" with a Stellar expression that evaluates to true. This allows you to avoid the expense of Threat Triage, in cases where you know it is not needed. The "is_alert" field is used as a flag to indicate which messages should undergo Threat Triage processing. Once a message completes Threat Triage and gets a score, changing or removing the "is_alert" field does not do anything. “
>>>>>>>” Also i pushed json data to the new datasource which i configured . I created a new elastic search template. When reading the readme file, it has been mentioned to create metron_alert field. What is the difference between is_alert and metron_alert.”
As explained earlier, the is_alert field is required on the incoming message/event from the telemetry source. If the field is present and set to true, then the event/message will be undergo threat triage processing and a score will be assigned to it using the rules defined. “metron_alert” field on the other hand is required on the index template on Elasticsearch (not on the incoming event/message). I stand to be corrected, but there are some references which mention that this is a dummy field (https://metron.apache.org/current-book/metron-platform/metron-elasticsearch/index.html)
>>>>>>>” But i couldn't find anything i. AlertsUI”
Usually it is because of a missing field that is required by Elasticsearch and metron. You may refer to documentation here https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch/metron-elasticsearch-common and https://docs.cloudera.com/HDPDocuments/HCP1/HCP-1.9.0/add-new-telemetry-data-source/content/create_elasticsearch_index_template.html
Refer to section “Elastic Search” and “Using Metron with ElasticSearch 5.6” for details. Please also refer to metron-rest logs. If you still face issues, please include your index template in your response.
Hope that helps.
Best regards,
Sanket
From: Geeks Girls <ge...@gmail.com>
Reply to: "user@metron.apache.org" <us...@metron.apache.org>
Date: Wednesday, 04 December 2019 at 21:34
To: "user@metron.apache.org" <us...@metron.apache.org>
Subject: Data not populating in metron alerts ui
Hi,
I am planning to use Metron as a SIEM and exploring it's features. Thanks for the great documentation. It helped a lot to set it up quickly. Initially configured snort ,bro,yaf logs to flow into Metron . For snort, could see threat triage rules configured in the Metron enrichment config. But for bro logs, is_alert field is blank .I verified the data in the Kibana. Though the is_alert is blank ,those logs also appearing in metron alerts ui.How this could be possible.
Also i pushed json data to the new datasource which i configured . I created a new elastic search template. When reading the readme file, it has been mentioned to create metron_alert field. What is the difference between is_alert and metron_alert.
What are the configuration needed to push data as alerts in metron AlertsUI. I could see logs are being parsed,enriched and indexed in the elastic search. So created Kibana dashboard .But i couldn't find anything i. AlertsUI. What should i do?Any help is highly appreciated.
Thanks,
Jai