You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by tkg_cangkul <yu...@gmail.com> on 2017/10/16 09:23:12 UTC
event correlation on metron
hi,
anyone could explain me about event correlation using apache metron?
does metron support event correlation?
Pls Advice
Fwd: event correlation on metron
Posted by Youzha <yu...@gmail.com>.
---------- Forwarded message ---------
From: Youzha <yu...@gmail.com>
Date: Tue, 17 Oct 2017 at 22.53
Subject: Re: event correlation on metron
To: <us...@metron.apache.org>
is it possible to ingest other logs like /var/log/secure for example to be
new telemetry on metron? i’ve seen the metron architecture on the website
like picture below. host logs, email, av, etc can be telemetry event buffer
on metron. if this possible, could you give me some suggestion how to do it
?
On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
> If you want to look at failed login attempts for each user over time, then
> the Profiler might be a good solution. Your profile will depend on the
> fields available in your telemetry, but it would look something like this,
> as an example.
>
> {
> "profile": "failed-logins",
> "foreach": "user.name",
> "onlyif": "source.type == 'activedirectory' and event.type ==
> 'failed_login'"
> "init": { "count": 0 },
> "update": { "count" : "count + 1" },
> "result": "count"
> }
>
>
> You can find an introduction and more information on using the Profiler
> below.
> *
> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>
> Best of luck
>
> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com>
> wrote:
>
>> for example,
>>
>> i wanna try to correlate between logs.
>> how many times user A have login failed and how many times user A have
>> login succeed. include detail IP, timestamp etc.
>> is this possible to do with metron?
>>
>>
>>
>>
>> On 17/10/17 02:56, James Sirota wrote:
>>
>>> What specifically are you looking to correlate? Can you talk a little
>>> more about your use case?
>>>
>>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>>
>>>> hi,
>>>>
>>>> anyone could explain me about event correlation using apache metron?
>>>> does metron support event correlation?
>>>>
>>>> Pls Advice
>>>>
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>
>>
>
Re: event correlation on metron
Posted by Youzha <yu...@gmail.com>.
is there any documentation to create new sensor in metron?
On Wed, 18 Oct 2017 at 01.22 Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> Best bet there is to create a new sensor config using the grok parser
> type. So you would for example have a kafka topic called host_dhcp and a
> sensor called host_dhcp with the relevant grok pattern.
>
> Simon
>
>
> On 17 Oct 2017, at 19:19, Youzha <yu...@gmail.com> wrote:
>
> that’s what i mean.
> what sensor that i need if i want to do this case?
> especially when i wanna parse some host logs into metron enrichment and
> indexing
>
> On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
>> What you want to do in this setting is just TailFile, the just push to
>> Kafka. The grok piece is more efficiently handled in the Metron grok parser.
>>
>> Push to a kafka topic named for your sensor, then setup a sensor (a
>> parser topology to do the grok parsing and any transformation you need).
>> Each sensor gets its own parser topology.
>>
>> Simon
>>
>>
>> On 17 Oct 2017, at 19:00, Youzha <yu...@gmail.com> wrote:
>>
>> after nifi procces :
>>
>> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
>>
>> what metron topology that i can use to procces the data in kafka? so it
>> can be enrichment by metron. i’ve check the article about adding new
>> telemetry source with squid, there is a squid topology that will ingest
>> from the squid topic in kafka and then put on enrichment kafka topic.
>> so how about my use case above? is there any topology that i can use?
>>
>> On Wed, 18 Oct 2017 at 00.30 Otto Fowler <ot...@gmail.com> wrote:
>>
>>> So,
>>> There are several options parsing the data and enriching.
>>>
>>> 1. A native parser ( java ), which you have noticed is not there
>>> 2. An instance of the GROK parser, with GROK rules that parser the input
>>> 3. If it is CSV an instance of the CSV parser
>>> 4. If it is JSON an instance of the JSONMap parser
>>>
>>> If these cannot be applied to your file then your options are:
>>>
>>> 1. Write or open a jira for a native parser
>>> 2. find a way to transform your data to one of the above formats, so you
>>> can use those parsers. This again is where nifi can help. Something like:
>>>
>>>
>>> [nifi]
>>>
>>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>>>
>>> where TRANSFORM_TO_JSON is a script processor or something built in
>>> depending on your format.
>>>
>>>
>>>
>>> On October 17, 2017 at 13:16:05, Youzha (yuza.rasfar@gmail.com) wrote:
>>>
>>> Hi Lauren thx for your reply,
>>>
>>> yeah your suggestion absolutely right. i was able to ingest the logs to
>>> kafka. but how metron can enrich and index all of it? i think there are
>>> only bro, snort, yaf, snort, pcap, websphere topology storm on metron for
>>> parsers. so, how metron can read the logs telemetry and proccess it so i
>>> can use it to event correlation
>>>
>>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets <la...@daemon.be> wrote:
>>>
>>>> Hi Youzha,
>>>>
>>>> Either check how the snort logs on the full dev installation are
>>>> ingested (I believe it's with a script) or check the Apache NiFi project
>>>> which makes it very easy to read logs from almost any format and ingest
>>>> them to Metron via Kafka.
>>>>
>>>> On 2017-10-17 08:53, Youzha wrote:
>>>>
>>>> is it possible to ingest other logs like /var/log/secure for example to
>>>> be new telemetry on metron? i've seen the metron architecture on the
>>>> website like picture below. host logs, email, av, etc can be telemetry
>>>> event buffer on metron. if this possible, could you give me some suggestion
>>>> how to do it ?
>>>>
>>>>
>>>> On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
>>>>
>>>>> If you want to look at failed login attempts for each user over time,
>>>>> then the Profiler might be a good solution. Your profile will depend on
>>>>> the fields available in your telemetry, but it would look something like
>>>>> this, as an example.
>>>>>
>>>>>
>>>>> {
>>>>> "profile": "failed-logins",
>>>>> "foreach": "user.name",
>>>>> "onlyif": "source.type == 'activedirectory' and event.type ==
>>>>> 'failed_login'"
>>>>> "init": { "count": 0 },
>>>>> "update": { "count" : "count + 1" },
>>>>> "result": "count"
>>>>> }
>>>>>
>>>>>
>>>>> You can find an introduction and more information on using the
>>>>> Profiler below.
>>>>> *
>>>>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>>>>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>>>>
>>>>> Best of luck
>>>>>
>>>>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> for example,
>>>>>>
>>>>>> i wanna try to correlate between logs.
>>>>>> how many times user A have login failed and how many times user A
>>>>>> have login succeed. include detail IP, timestamp etc.
>>>>>> is this possible to do with metron?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 17/10/17 02:56, James Sirota wrote:
>>>>>>
>>>>>>> What specifically are you looking to correlate? Can you talk a
>>>>>>> little more about your use case?
>>>>>>>
>>>>>>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>>>>>>
>>>>>>>> hi,
>>>>>>>>
>>>>>>>> anyone could explain me about event correlation using apache metron?
>>>>>>>> does metron support event correlation?
>>>>>>>>
>>>>>>>> Pls Advice
>>>>>>>
>>>>>>> -------------------
>>>>>>> Thank you,
>>>>>>>
>>>>>>> James Sirota
>>>>>>> PMC- Apache Metron
>>>>>>> jsirota AT apache DOT org
>>>>>>
>>>>>>
>>>>
Re: event correlation on metron
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Best bet there is to create a new sensor config using the grok parser type. So you would for example have a kafka topic called host_dhcp and a sensor called host_dhcp with the relevant grok pattern.
Simon
> On 17 Oct 2017, at 19:19, Youzha <yu...@gmail.com> wrote:
>
> that’s what i mean.
> what sensor that i need if i want to do this case?
> especially when i wanna parse some host logs into metron enrichment and indexing
>
>> On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball <si...@simonellistonball.com> wrote:
>> What you want to do in this setting is just TailFile, the just push to Kafka. The grok piece is more efficiently handled in the Metron grok parser.
>>
>> Push to a kafka topic named for your sensor, then setup a sensor (a parser topology to do the grok parsing and any transformation you need). Each sensor gets its own parser topology.
>>
>> Simon
>>
>>
>>> On 17 Oct 2017, at 19:00, Youzha <yu...@gmail.com> wrote:
>>>
>>> after nifi procces :
>>>
>>> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
>>>
>>> what metron topology that i can use to procces the data in kafka? so it can be enrichment by metron. i’ve check the article about adding new telemetry source with squid, there is a squid topology that will ingest from the squid topic in kafka and then put on enrichment kafka topic.
>>> so how about my use case above? is there any topology that i can use?
>>>
>>>> On Wed, 18 Oct 2017 at 00.30 Otto Fowler <ot...@gmail.com> wrote:
>>>> So,
>>>> There are several options parsing the data and enriching.
>>>>
>>>> 1. A native parser ( java ), which you have noticed is not there
>>>> 2. An instance of the GROK parser, with GROK rules that parser the input
>>>> 3. If it is CSV an instance of the CSV parser
>>>> 4. If it is JSON an instance of the JSONMap parser
>>>>
>>>> If these cannot be applied to your file then your options are:
>>>>
>>>> 1. Write or open a jira for a native parser
>>>> 2. find a way to transform your data to one of the above formats, so you can use those parsers. This again is where nifi can help. Something like:
>>>>
>>>>
>>>> [nifi]
>>>>
>>>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>>>>
>>>> where TRANSFORM_TO_JSON is a script processor or something built in depending on your format.
>>>>
>>>>
>>>>
>>>>> On October 17, 2017 at 13:16:05, Youzha (yuza.rasfar@gmail.com) wrote:
>>>>>
>>>>> Hi Lauren thx for your reply,
>>>>>
>>>>> yeah your suggestion absolutely right. i was able to ingest the logs to kafka. but how metron can enrich and index all of it? i think there are only bro, snort, yaf, snort, pcap, websphere topology storm on metron for parsers. so, how metron can read the logs telemetry and proccess it so i can use it to event correlation
>>>>>
>>>>>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets <la...@daemon.be> wrote:
>>>>>> Hi Youzha,
>>>>>>
>>>>>> Either check how the snort logs on the full dev installation are ingested (I believe it's with a script) or check the Apache NiFi project which makes it very easy to read logs from almost any format and ingest them to Metron via Kafka.
>>>>>>
>>>>>>> On 2017-10-17 08:53, Youzha wrote:
>>>>>>>
>>>>>>> is it possible to ingest other logs like /var/log/secure for example to be new telemetry on metron? i've seen the metron architecture on the website like picture below. host logs, email, av, etc can be telemetry event buffer on metron. if this possible, could you give me some suggestion how to do it ?
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
>>>>>>>> If you want to look at failed login attempts for each user over time, then the Profiler might be a good solution. Your profile will depend on the fields available in your telemetry, but it would look something like this, as an example.
>>>>>>>>
>>>>>>>> {
>>>>>>>> "profile": "failed-logins",
>>>>>>>> "foreach": "user.name",
>>>>>>>> "onlyif": "source.type == 'activedirectory' and event.type == 'failed_login'"
>>>>>>>> "init": { "count": 0 },
>>>>>>>> "update": { "count" : "count + 1" },
>>>>>>>> "result": "count"
>>>>>>>> }
>>>>>>>>
>>>>>>>> You can find an introduction and more information on using the Profiler below.
>>>>>>>> * https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>>>>>>>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>>>>>>>
>>>>>>>> Best of luck
>>>>>>>>
>>>>>>>>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com> wrote:
>>>>>>>>> for example,
>>>>>>>>>
>>>>>>>>> i wanna try to correlate between logs.
>>>>>>>>> how many times user A have login failed and how many times user A have login succeed. include detail IP, timestamp etc.
>>>>>>>>> is this possible to do with metron?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 17/10/17 02:56, James Sirota wrote:
>>>>>>>>>> What specifically are you looking to correlate? Can you talk a little more about your use case?
>>>>>>>>>>
>>>>>>>>>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>>>>>>>>>> hi,
>>>>>>>>>>>
>>>>>>>>>>> anyone could explain me about event correlation using apache metron?
>>>>>>>>>>> does metron support event correlation?
>>>>>>>>>>>
>>>>>>>>>>> Pls Advice
>>>>>>>>>> -------------------
>>>>>>>>>> Thank you,
>>>>>>>>>>
>>>>>>>>>> James Sirota
>>>>>>>>>> PMC- Apache Metron
>>>>>>>>>> jsirota AT apache DOT org
>>>>>>
Re: event correlation on metron
Posted by Youzha <yu...@gmail.com>.
that’s what i mean.
what sensor that i need if i want to do this case?
especially when i wanna parse some host logs into metron enrichment and
indexing
On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> What you want to do in this setting is just TailFile, the just push to
> Kafka. The grok piece is more efficiently handled in the Metron grok parser.
>
> Push to a kafka topic named for your sensor, then setup a sensor (a parser
> topology to do the grok parsing and any transformation you need). Each
> sensor gets its own parser topology.
>
> Simon
>
>
> On 17 Oct 2017, at 19:00, Youzha <yu...@gmail.com> wrote:
>
> after nifi procces :
>
> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
>
> what metron topology that i can use to procces the data in kafka? so it
> can be enrichment by metron. i’ve check the article about adding new
> telemetry source with squid, there is a squid topology that will ingest
> from the squid topic in kafka and then put on enrichment kafka topic.
> so how about my use case above? is there any topology that i can use?
>
> On Wed, 18 Oct 2017 at 00.30 Otto Fowler <ot...@gmail.com> wrote:
>
>> So,
>> There are several options parsing the data and enriching.
>>
>> 1. A native parser ( java ), which you have noticed is not there
>> 2. An instance of the GROK parser, with GROK rules that parser the input
>> 3. If it is CSV an instance of the CSV parser
>> 4. If it is JSON an instance of the JSONMap parser
>>
>> If these cannot be applied to your file then your options are:
>>
>> 1. Write or open a jira for a native parser
>> 2. find a way to transform your data to one of the above formats, so you
>> can use those parsers. This again is where nifi can help. Something like:
>>
>>
>> [nifi]
>>
>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>>
>> where TRANSFORM_TO_JSON is a script processor or something built in
>> depending on your format.
>>
>>
>>
>> On October 17, 2017 at 13:16:05, Youzha (yuza.rasfar@gmail.com) wrote:
>>
>> Hi Lauren thx for your reply,
>>
>> yeah your suggestion absolutely right. i was able to ingest the logs to
>> kafka. but how metron can enrich and index all of it? i think there are
>> only bro, snort, yaf, snort, pcap, websphere topology storm on metron for
>> parsers. so, how metron can read the logs telemetry and proccess it so i
>> can use it to event correlation
>>
>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets <la...@daemon.be> wrote:
>>
>>> Hi Youzha,
>>>
>>> Either check how the snort logs on the full dev installation are
>>> ingested (I believe it's with a script) or check the Apache NiFi project
>>> which makes it very easy to read logs from almost any format and ingest
>>> them to Metron via Kafka.
>>>
>>> On 2017-10-17 08:53, Youzha wrote:
>>>
>>> is it possible to ingest other logs like /var/log/secure for example to
>>> be new telemetry on metron? i've seen the metron architecture on the
>>> website like picture below. host logs, email, av, etc can be telemetry
>>> event buffer on metron. if this possible, could you give me some suggestion
>>> how to do it ?
>>>
>>>
>>> On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
>>>
>>>> If you want to look at failed login attempts for each user over time,
>>>> then the Profiler might be a good solution. Your profile will depend on
>>>> the fields available in your telemetry, but it would look something like
>>>> this, as an example.
>>>>
>>>>
>>>> {
>>>> "profile": "failed-logins",
>>>> "foreach": "user.name",
>>>> "onlyif": "source.type == 'activedirectory' and event.type ==
>>>> 'failed_login'"
>>>> "init": { "count": 0 },
>>>> "update": { "count" : "count + 1" },
>>>> "result": "count"
>>>> }
>>>>
>>>>
>>>> You can find an introduction and more information on using the Profiler
>>>> below.
>>>> *
>>>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>>>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>>>
>>>> Best of luck
>>>>
>>>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com>
>>>> wrote:
>>>>
>>>>> for example,
>>>>>
>>>>> i wanna try to correlate between logs.
>>>>> how many times user A have login failed and how many times user A have
>>>>> login succeed. include detail IP, timestamp etc.
>>>>> is this possible to do with metron?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 17/10/17 02:56, James Sirota wrote:
>>>>>
>>>>>> What specifically are you looking to correlate? Can you talk a
>>>>>> little more about your use case?
>>>>>>
>>>>>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>>>>>
>>>>>>> hi,
>>>>>>>
>>>>>>> anyone could explain me about event correlation using apache metron?
>>>>>>> does metron support event correlation?
>>>>>>>
>>>>>>> Pls Advice
>>>>>>
>>>>>> -------------------
>>>>>> Thank you,
>>>>>>
>>>>>> James Sirota
>>>>>> PMC- Apache Metron
>>>>>> jsirota AT apache DOT org
>>>>>
>>>>>
>>>
Re: event correlation on metron
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
What you want to do in this setting is just TailFile, the just push to Kafka. The grok piece is more efficiently handled in the Metron grok parser.
Push to a kafka topic named for your sensor, then setup a sensor (a parser topology to do the grok parsing and any transformation you need). Each sensor gets its own parser topology.
Simon
> On 17 Oct 2017, at 19:00, Youzha <yu...@gmail.com> wrote:
>
> after nifi procces :
>
> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
>
> what metron topology that i can use to procces the data in kafka? so it can be enrichment by metron. i’ve check the article about adding new telemetry source with squid, there is a squid topology that will ingest from the squid topic in kafka and then put on enrichment kafka topic.
> so how about my use case above? is there any topology that i can use?
>
>> On Wed, 18 Oct 2017 at 00.30 Otto Fowler <ot...@gmail.com> wrote:
>> So,
>> There are several options parsing the data and enriching.
>>
>> 1. A native parser ( java ), which you have noticed is not there
>> 2. An instance of the GROK parser, with GROK rules that parser the input
>> 3. If it is CSV an instance of the CSV parser
>> 4. If it is JSON an instance of the JSONMap parser
>>
>> If these cannot be applied to your file then your options are:
>>
>> 1. Write or open a jira for a native parser
>> 2. find a way to transform your data to one of the above formats, so you can use those parsers. This again is where nifi can help. Something like:
>>
>>
>> [nifi]
>>
>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>>
>> where TRANSFORM_TO_JSON is a script processor or something built in depending on your format.
>>
>>
>>
>>> On October 17, 2017 at 13:16:05, Youzha (yuza.rasfar@gmail.com) wrote:
>>>
>>> Hi Lauren thx for your reply,
>>>
>>> yeah your suggestion absolutely right. i was able to ingest the logs to kafka. but how metron can enrich and index all of it? i think there are only bro, snort, yaf, snort, pcap, websphere topology storm on metron for parsers. so, how metron can read the logs telemetry and proccess it so i can use it to event correlation
>>>
>>>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets <la...@daemon.be> wrote:
>>>> Hi Youzha,
>>>>
>>>> Either check how the snort logs on the full dev installation are ingested (I believe it's with a script) or check the Apache NiFi project which makes it very easy to read logs from almost any format and ingest them to Metron via Kafka.
>>>>
>>>>> On 2017-10-17 08:53, Youzha wrote:
>>>>>
>>>>> is it possible to ingest other logs like /var/log/secure for example to be new telemetry on metron? i've seen the metron architecture on the website like picture below. host logs, email, av, etc can be telemetry event buffer on metron. if this possible, could you give me some suggestion how to do it ?
>>>>>
>>>>>
>>>>>> On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
>>>>>> If you want to look at failed login attempts for each user over time, then the Profiler might be a good solution. Your profile will depend on the fields available in your telemetry, but it would look something like this, as an example.
>>>>>>
>>>>>> {
>>>>>> "profile": "failed-logins",
>>>>>> "foreach": "user.name",
>>>>>> "onlyif": "source.type == 'activedirectory' and event.type == 'failed_login'"
>>>>>> "init": { "count": 0 },
>>>>>> "update": { "count" : "count + 1" },
>>>>>> "result": "count"
>>>>>> }
>>>>>>
>>>>>> You can find an introduction and more information on using the Profiler below.
>>>>>> * https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>>>>>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>>>>>
>>>>>> Best of luck
>>>>>>
>>>>>>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com> wrote:
>>>>>>> for example,
>>>>>>>
>>>>>>> i wanna try to correlate between logs.
>>>>>>> how many times user A have login failed and how many times user A have login succeed. include detail IP, timestamp etc.
>>>>>>> is this possible to do with metron?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 17/10/17 02:56, James Sirota wrote:
>>>>>>>> What specifically are you looking to correlate? Can you talk a little more about your use case?
>>>>>>>>
>>>>>>>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>>>>>>>> hi,
>>>>>>>>>
>>>>>>>>> anyone could explain me about event correlation using apache metron?
>>>>>>>>> does metron support event correlation?
>>>>>>>>>
>>>>>>>>> Pls Advice
>>>>>>>> -------------------
>>>>>>>> Thank you,
>>>>>>>>
>>>>>>>> James Sirota
>>>>>>>> PMC- Apache Metron
>>>>>>>> jsirota AT apache DOT org
>>>>
Re: event correlation on metron
Posted by Youzha <yu...@gmail.com>.
after nifi procces :
TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
what metron topology that i can use to procces the data in kafka? so it can
be enrichment by metron. i’ve check the article about adding new telemetry
source with squid, there is a squid topology that will ingest from the
squid topic in kafka and then put on enrichment kafka topic.
so how about my use case above? is there any topology that i can use?
On Wed, 18 Oct 2017 at 00.30 Otto Fowler <ot...@gmail.com> wrote:
> So,
> There are several options parsing the data and enriching.
>
> 1. A native parser ( java ), which you have noticed is not there
> 2. An instance of the GROK parser, with GROK rules that parser the input
> 3. If it is CSV an instance of the CSV parser
> 4. If it is JSON an instance of the JSONMap parser
>
> If these cannot be applied to your file then your options are:
>
> 1. Write or open a jira for a native parser
> 2. find a way to transform your data to one of the above formats, so you
> can use those parsers. This again is where nifi can help. Something like:
>
>
> [nifi]
>
> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>
> where TRANSFORM_TO_JSON is a script processor or something built in
> depending on your format.
>
>
>
> On October 17, 2017 at 13:16:05, Youzha (yuza.rasfar@gmail.com) wrote:
>
> Hi Lauren thx for your reply,
>
> yeah your suggestion absolutely right. i was able to ingest the logs to
> kafka. but how metron can enrich and index all of it? i think there are
> only bro, snort, yaf, snort, pcap, websphere topology storm on metron for
> parsers. so, how metron can read the logs telemetry and proccess it so i
> can use it to event correlation
>
> On Tue, 17 Oct 2017 at 23.11 Laurens Vets <la...@daemon.be> wrote:
>
>> Hi Youzha,
>>
>> Either check how the snort logs on the full dev installation are ingested
>> (I believe it's with a script) or check the Apache NiFi project which makes
>> it very easy to read logs from almost any format and ingest them to Metron
>> via Kafka.
>>
>> On 2017-10-17 08:53, Youzha wrote:
>>
>> is it possible to ingest other logs like /var/log/secure for example to
>> be new telemetry on metron? i've seen the metron architecture on the
>> website like picture below. host logs, email, av, etc can be telemetry
>> event buffer on metron. if this possible, could you give me some suggestion
>> how to do it ?
>>
>>
>> On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
>>
>>> If you want to look at failed login attempts for each user over time,
>>> then the Profiler might be a good solution. Your profile will depend on
>>> the fields available in your telemetry, but it would look something like
>>> this, as an example.
>>>
>>>
>>> {
>>> "profile": "failed-logins",
>>> "foreach": "user.name",
>>> "onlyif": "source.type == 'activedirectory' and event.type ==
>>> 'failed_login'"
>>> "init": { "count": 0 },
>>> "update": { "count" : "count + 1" },
>>> "result": "count"
>>> }
>>>
>>>
>>> You can find an introduction and more information on using the Profiler
>>> below.
>>> *
>>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>>
>>> Best of luck
>>>
>>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com>
>>> wrote:
>>>
>>>> for example,
>>>>
>>>> i wanna try to correlate between logs.
>>>> how many times user A have login failed and how many times user A have
>>>> login succeed. include detail IP, timestamp etc.
>>>> is this possible to do with metron?
>>>>
>>>>
>>>>
>>>>
>>>> On 17/10/17 02:56, James Sirota wrote:
>>>>
>>>>> What specifically are you looking to correlate? Can you talk a little
>>>>> more about your use case?
>>>>>
>>>>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>>>>
>>>>>> hi,
>>>>>>
>>>>>> anyone could explain me about event correlation using apache metron?
>>>>>> does metron support event correlation?
>>>>>>
>>>>>> Pls Advice
>>>>>
>>>>> -------------------
>>>>> Thank you,
>>>>>
>>>>> James Sirota
>>>>> PMC- Apache Metron
>>>>> jsirota AT apache DOT org
>>>>
>>>>
>>
Re: event correlation on metron
Posted by ed d <ra...@hotmail.com>.
is there a guide of sorts we can follow, or noodle through, to write our own java based parser?
or do we need to just java through and figure it out?
________________________________
From: Otto Fowler <ot...@gmail.com>
Sent: Tuesday, October 17, 2017 1:30 PM
To: Youzha; user@metron.apache.org
Subject: Re: event correlation on metron
So,
There are several options parsing the data and enriching.
1. A native parser ( java ), which you have noticed is not there
2. An instance of the GROK parser, with GROK rules that parser the input
3. If it is CSV an instance of the CSV parser
4. If it is JSON an instance of the JSONMap parser
If these cannot be applied to your file then your options are:
1. Write or open a jira for a native parser
2. find a way to transform your data to one of the above formats, so you can use those parsers. This again is where nifi can help. Something like:
[nifi]
TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
where TRANSFORM_TO_JSON is a script processor or something built in depending on your format.
On October 17, 2017 at 13:16:05, Youzha (yuza.rasfar@gmail.com<ma...@gmail.com>) wrote:
Hi Lauren thx for your reply,
yeah your suggestion absolutely right. i was able to ingest the logs to kafka. but how metron can enrich and index all of it? i think there are only bro, snort, yaf, snort, pcap, websphere topology storm on metron for parsers. so, how metron can read the logs telemetry and proccess it so i can use it to event correlation
On Tue, 17 Oct 2017 at 23.11 Laurens Vets <la...@daemon.be>> wrote:
Hi Youzha,
Either check how the snort logs on the full dev installation are ingested (I believe it's with a script) or check the Apache NiFi project which makes it very easy to read logs from almost any format and ingest them to Metron via Kafka.
On 2017-10-17 08:53, Youzha wrote:
is it possible to ingest other logs like /var/log/secure for example to be new telemetry on metron? i've seen the metron architecture on the website like picture below. host logs, email, av, etc can be telemetry event buffer on metron. if this possible, could you give me some suggestion how to do it ?
[X]
On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org>> wrote:
If you want to look at failed login attempts for each user over time, then the Profiler might be a good solution. Your profile will depend on the fields available in your telemetry, but it would look something like this, as an example.
{
"profile": "failed-logins",
"foreach": "user.name<https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fuser.name&data=02%7C01%7Cragdelaed%40hotmail.com%7Cfff6af2580d14dbe7cc108d51584d773%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636438582643979156&sdata=CAtDY7OiqSOr4217Br0AAS7oG2GBtd6kHU8wgxvFn4s%3D&reserved=0>",
"onlyif": "source.type == 'activedirectory' and event.type == 'failed_login'"
"init": { "count": 0 },
"update": { "count" : "count + 1" },
"result": "count"
}
You can find an introduction and more information on using the Profiler below.
* https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fmetron%2Ftree%2Fmaster%2Fmetron-analytics%2Fmetron-profiler&data=02%7C01%7Cragdelaed%40hotmail.com%7Cfff6af2580d14dbe7cc108d51584d773%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636438582643979156&sdata=yqV3Vo%2FA766aAh5wfeCWX%2BH%2BhqQTMh4zveuIV%2BtJMJU%3D&reserved=0>
* https://www.slideshare.net/secret/GFBf2RTXBG35PB<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.slideshare.net%2Fsecret%2FGFBf2RTXBG35PB&data=02%7C01%7Cragdelaed%40hotmail.com%7Cfff6af2580d14dbe7cc108d51584d773%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636438582643979156&sdata=9linbLpi2yvYFV4jtcA4sK5neMIggpGr4Lc2c%2BaGTwo%3D&reserved=0>
Best of luck
On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com>> wrote:
for example,
i wanna try to correlate between logs.
how many times user A have login failed and how many times user A have login succeed. include detail IP, timestamp etc.
is this possible to do with metron?
On 17/10/17 02:56, James Sirota wrote:
What specifically are you looking to correlate? Can you talk a little more about your use case?
16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>>:
hi,
anyone could explain me about event correlation using apache metron?
does metron support event correlation?
Pls Advice
-------------------
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
Re: event correlation on metron
Posted by Otto Fowler <ot...@gmail.com>.
So,
There are several options parsing the data and enriching.
1. A native parser ( java ), which you have noticed is not there
2. An instance of the GROK parser, with GROK rules that parser the input
3. If it is CSV an instance of the CSV parser
4. If it is JSON an instance of the JSONMap parser
If these cannot be applied to your file then your options are:
1. Write or open a jira for a native parser
2. find a way to transform your data to one of the above formats, so you
can use those parsers. This again is where nifi can help. Something like:
[nifi]
TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
where TRANSFORM_TO_JSON is a script processor or something built in
depending on your format.
On October 17, 2017 at 13:16:05, Youzha (yuza.rasfar@gmail.com) wrote:
Hi Lauren thx for your reply,
yeah your suggestion absolutely right. i was able to ingest the logs to
kafka. but how metron can enrich and index all of it? i think there are
only bro, snort, yaf, snort, pcap, websphere topology storm on metron for
parsers. so, how metron can read the logs telemetry and proccess it so i
can use it to event correlation
On Tue, 17 Oct 2017 at 23.11 Laurens Vets <la...@daemon.be> wrote:
> Hi Youzha,
>
> Either check how the snort logs on the full dev installation are ingested
> (I believe it's with a script) or check the Apache NiFi project which makes
> it very easy to read logs from almost any format and ingest them to Metron
> via Kafka.
>
> On 2017-10-17 08:53, Youzha wrote:
>
> is it possible to ingest other logs like /var/log/secure for example to be
> new telemetry on metron? i've seen the metron architecture on the website
> like picture below. host logs, email, av, etc can be telemetry event buffer
> on metron. if this possible, could you give me some suggestion how to do it
> ?
>
>
> On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
>
>> If you want to look at failed login attempts for each user over time,
>> then the Profiler might be a good solution. Your profile will depend on
>> the fields available in your telemetry, but it would look something like
>> this, as an example.
>>
>>
>> {
>> "profile": "failed-logins",
>> "foreach": "user.name",
>> "onlyif": "source.type == 'activedirectory' and event.type ==
>> 'failed_login'"
>> "init": { "count": 0 },
>> "update": { "count" : "count + 1" },
>> "result": "count"
>> }
>>
>>
>> You can find an introduction and more information on using the Profiler
>> below.
>> *
>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>
>> Best of luck
>>
>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com>
>> wrote:
>>
>>> for example,
>>>
>>> i wanna try to correlate between logs.
>>> how many times user A have login failed and how many times user A have
>>> login succeed. include detail IP, timestamp etc.
>>> is this possible to do with metron?
>>>
>>>
>>>
>>>
>>> On 17/10/17 02:56, James Sirota wrote:
>>>
>>>> What specifically are you looking to correlate? Can you talk a little
>>>> more about your use case?
>>>>
>>>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>>>
>>>>> hi,
>>>>>
>>>>> anyone could explain me about event correlation using apache metron?
>>>>> does metron support event correlation?
>>>>>
>>>>> Pls Advice
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>
>>>
>
Re: event correlation on metron
Posted by Youzha <yu...@gmail.com>.
Hi Lauren thx for your reply,
yeah your suggestion absolutely right. i was able to ingest the logs to
kafka. but how metron can enrich and index all of it? i think there are
only bro, snort, yaf, snort, pcap, websphere topology storm on metron for
parsers. so, how metron can read the logs telemetry and proccess it so i
can use it to event correlation
On Tue, 17 Oct 2017 at 23.11 Laurens Vets <la...@daemon.be> wrote:
> Hi Youzha,
>
> Either check how the snort logs on the full dev installation are ingested
> (I believe it's with a script) or check the Apache NiFi project which makes
> it very easy to read logs from almost any format and ingest them to Metron
> via Kafka.
>
> On 2017-10-17 08:53, Youzha wrote:
>
> is it possible to ingest other logs like /var/log/secure for example to be
> new telemetry on metron? i've seen the metron architecture on the website
> like picture below. host logs, email, av, etc can be telemetry event buffer
> on metron. if this possible, could you give me some suggestion how to do it
> ?
>
>
> On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
>
>> If you want to look at failed login attempts for each user over time,
>> then the Profiler might be a good solution. Your profile will depend on
>> the fields available in your telemetry, but it would look something like
>> this, as an example.
>>
>>
>> {
>> "profile": "failed-logins",
>> "foreach": "user.name",
>> "onlyif": "source.type == 'activedirectory' and event.type ==
>> 'failed_login'"
>> "init": { "count": 0 },
>> "update": { "count" : "count + 1" },
>> "result": "count"
>> }
>>
>>
>> You can find an introduction and more information on using the Profiler
>> below.
>> *
>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>
>> Best of luck
>>
>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com>
>> wrote:
>>
>>> for example,
>>>
>>> i wanna try to correlate between logs.
>>> how many times user A have login failed and how many times user A have
>>> login succeed. include detail IP, timestamp etc.
>>> is this possible to do with metron?
>>>
>>>
>>>
>>>
>>> On 17/10/17 02:56, James Sirota wrote:
>>>
>>>> What specifically are you looking to correlate? Can you talk a little
>>>> more about your use case?
>>>>
>>>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>>>
>>>>> hi,
>>>>>
>>>>> anyone could explain me about event correlation using apache metron?
>>>>> does metron support event correlation?
>>>>>
>>>>> Pls Advice
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>
>>>
>
Re: event correlation on metron
Posted by Laurens Vets <la...@daemon.be>.
Hi Youzha,
Either check how the snort logs on the full dev installation are
ingested (I believe it's with a script) or check the Apache NiFi project
which makes it very easy to read logs from almost any format and ingest
them to Metron via Kafka.
On 2017-10-17 08:53, Youzha wrote:
> is it possible to ingest other logs like /var/log/secure for example to be new telemetry on metron? i've seen the metron architecture on the website like picture below. host logs, email, av, etc can be telemetry event buffer on metron. if this possible, could you give me some suggestion how to do it ?
>
> On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
>
> If you want to look at failed login attempts for each user over time, then the Profiler might be a good solution. Your profile will depend on the fields available in your telemetry, but it would look something like this, as an example.
>
> {
>
> "profile": "failed-logins",
>
> "foreach": "user.name [1]",
>
> "onlyif": "source.type == 'activedirectory' and event.type == 'failed_login'"
>
> "init": { "count": 0 },
>
> "update": { "count" : "count + 1" },
>
> "result": "count"
>
> }
>
> You can find an introduction and more information on using the Profiler below.
> * https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>
> Best of luck
>
> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com> wrote:
> for example,
>
> i wanna try to correlate between logs.
> how many times user A have login failed and how many times user A have login succeed. include detail IP, timestamp etc.
> is this possible to do with metron?
>
> On 17/10/17 02:56, James Sirota wrote:
> What specifically are you looking to correlate? Can you talk a little more about your use case?
>
> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
> hi,
>
> anyone could explain me about event correlation using apache metron?
> does metron support event correlation?
>
> Pls Advice -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
Links:
------
[1] http://user.name
Re: event correlation on metron
Posted by Youzha <yu...@gmail.com>.
is it possible to ingest other logs like /var/log/secure for example to be
new telemetry on metron? i’ve seen the metron architecture on the website
like picture below. host logs, email, av, etc can be telemetry event buffer
on metron. if this possible, could you give me some suggestion how to do it
?
On Tue, 17 Oct 2017 at 21.00 Nick Allen <ni...@nickallen.org> wrote:
> If you want to look at failed login attempts for each user over time, then
> the Profiler might be a good solution. Your profile will depend on the
> fields available in your telemetry, but it would look something like this,
> as an example.
>
> {
> "profile": "failed-logins",
> "foreach": "user.name",
> "onlyif": "source.type == 'activedirectory' and event.type ==
> 'failed_login'"
> "init": { "count": 0 },
> "update": { "count" : "count + 1" },
> "result": "count"
> }
>
>
> You can find an introduction and more information on using the Profiler
> below.
> *
> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>
> Best of luck
>
> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com>
> wrote:
>
>> for example,
>>
>> i wanna try to correlate between logs.
>> how many times user A have login failed and how many times user A have
>> login succeed. include detail IP, timestamp etc.
>> is this possible to do with metron?
>>
>>
>>
>>
>> On 17/10/17 02:56, James Sirota wrote:
>>
>>> What specifically are you looking to correlate? Can you talk a little
>>> more about your use case?
>>>
>>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>>
>>>> hi,
>>>>
>>>> anyone could explain me about event correlation using apache metron?
>>>> does metron support event correlation?
>>>>
>>>> Pls Advice
>>>>
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>
>>
>
Re: event correlation on metron
Posted by Nick Allen <ni...@nickallen.org>.
If you want to look at failed login attempts for each user over time, then
the Profiler might be a good solution. Your profile will depend on the
fields available in your telemetry, but it would look something like this,
as an example.
{
"profile": "failed-logins",
"foreach": "user.name",
"onlyif": "source.type == 'activedirectory' and event.type ==
'failed_login'"
"init": { "count": 0 },
"update": { "count" : "count + 1" },
"result": "count"
}
You can find an introduction and more information on using the Profiler
below.
*
https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
* https://www.slideshare.net/secret/GFBf2RTXBG35PB
Best of luck
On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul <yu...@gmail.com> wrote:
> for example,
>
> i wanna try to correlate between logs.
> how many times user A have login failed and how many times user A have
> login succeed. include detail IP, timestamp etc.
> is this possible to do with metron?
>
>
>
>
> On 17/10/17 02:56, James Sirota wrote:
>
>> What specifically are you looking to correlate? Can you talk a little
>> more about your use case?
>>
>> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>>
>>> hi,
>>>
>>> anyone could explain me about event correlation using apache metron?
>>> does metron support event correlation?
>>>
>>> Pls Advice
>>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>
>
Re: event correlation on metron
Posted by tkg_cangkul <yu...@gmail.com>.
for example,
i wanna try to correlate between logs.
how many times user A have login failed and how many times user A have
login succeed. include detail IP, timestamp etc.
is this possible to do with metron?
On 17/10/17 02:56, James Sirota wrote:
> What specifically are you looking to correlate? Can you talk a little more about your use case?
>
> 16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
>> hi,
>>
>> anyone could explain me about event correlation using apache metron?
>> does metron support event correlation?
>>
>> Pls Advice
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
Re: event correlation on metron
Posted by James Sirota <js...@apache.org>.
What specifically are you looking to correlate? Can you talk a little more about your use case?
16.10.2017, 02:23, "tkg_cangkul" <yu...@gmail.com>:
> hi,
>
> anyone could explain me about event correlation using apache metron?
> does metron support event correlation?
>
> Pls Advice
-------------------
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org