You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Sanket Sharma <sa...@dukstra.com> on 2019/03/04 09:28:44 UTC
Use case question
Hi,
I've been looking at metron for a few days now and I have a unique use - thought of asking the experts if it makes sense to use metron in this scenario.
My understanding of the project so far is that its a framework built for analyzing cybersecurity threats. This includes analyzing IP packets, network traffics, URLs etc to calculate risk scores etc. The framework also enables data scientists to build and test their models. There are data collection plugins that collect data from variety of sources, stream it over kafka and makes them available for use by various models.
Now, we have a customer facing portal where customers login, submit all kinds of orders and transactions. We were looking at ways to analyze fraud that originates from our portal and I stumbled upon Metron. While we can definitely use Metron for analyzing source traffic, but would it be a good idea to use Metron to analyze the actual transactions themselves? I do understand that we will have to build our models etc. but given that all the heavy lifting is already done, I'm tempted to try Metron for this use case (instead of re-inventing the wheel).
Is this possible/recommended? Or would you recommend using Metron strictly for network related analysis?
Best Regards,
Sanket
Re: Use case question
Posted by Michael Miklavcic <mi...@gmail.com>.
Sanket, you should definitely be able to use Metron for what you've
described. Here are some examples that you might find useful for comparison
- https://github.com/apache/metron/tree/master/use-cases
Best,
Mike
On Mon, Mar 4, 2019 at 5:24 AM Sanket Sharma <sa...@dukstra.com>
wrote:
> Hi Simon,
>
> Thank you for the quick response. The use case is not completely
> interactive to be honest. The users can submit transactions or batches and
> they can be "accepted" in the UI. We are then looking at running them them
> through the detection system and before they are finally submitted to the
> processing system. We have a bit of a freedom in what we do once the
> transactions/orders have been received.
>
> I think that does give me enough confidence to go ahead and prototype a
> solution. Thank you once again.
>
> I'm busy deploying the local vagrant image and will keep you posted of the
> findings.
>
> Best regards,
> Sanket
> ------------------------------
> *From:* Simon Elliston Ball <si...@simonellistonball.com>
> *Sent:* Monday, March 4, 2019 12:25 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Use case question
>
> Hi Sanket,
>
> This is certainly an interesting case. Metron is deliberately designed for
> flexibility in terms of ingest and schema, so that non-network data sources
> and use cases can be accommodated. The one caveat I would suggest is that
> the Metron pipeline is designed for analytics and detection, but not
> necessarily for the kind of guaranteed latency you might need for something
> like a web application experience. While it is streaming and realtime by
> nature, it can in some circumstances take a second or so to get a message
> from end to end, particularly if you have a lot of detection or models
> running, so it's not ideal as part of an interactive process. That said,
> for the actual detection of fraud, and strange behaviour patterns on your
> website, it would be a great fit.
>
> Hope that helps,
> Simon
>
> On Mon, 4 Mar 2019 at 02:04, Hammad <ha...@gmail.com> wrote:
>
> Following!!
>
> On Mon, Mar 4, 2019 at 2:29 PM Sanket Sharma <sa...@dukstra.com>
> wrote:
>
>
>
> Hi,
>
> I've been looking at metron for a few days now and I have a unique use -
> thought of asking the experts if it makes sense to use metron in this
> scenario.
>
> My understanding of the project so far is that its a framework built for
> analyzing cybersecurity threats. This includes analyzing IP packets,
> network traffics, URLs etc to calculate risk scores etc. The framework also
> enables data scientists to build and test their models. There are data
> collection plugins that collect data from variety of sources, stream it
> over kafka and makes them available for use by various models.
>
> Now, we have a customer facing portal where customers login, submit all
> kinds of orders and transactions. We were looking at ways to analyze fraud
> that originates from our portal and I stumbled upon Metron. While we can
> definitely use Metron for analyzing source traffic, but would it be a good
> idea to use Metron to analyze the actual transactions themselves? I do
> understand that we will have to build our models etc. but given that all
> the heavy lifting is already done, I'm tempted to try Metron for this use
> case (instead of re-inventing the wheel).
>
> Is this possible/recommended? Or would you recommend using Metron strictly
> for network related analysis?
>
> Best Regards,
> Sanket
>
>
>
> --
> --
> simon elliston ball
> @sireb
>
Re: Use case question
Posted by Sanket Sharma <sa...@dukstra.com>.
Hi Simon,
Thank you for the quick response. The use case is not completely interactive to be honest. The users can submit transactions or batches and they can be "accepted" in the UI. We are then looking at running them them through the detection system and before they are finally submitted to the processing system. We have a bit of a freedom in what we do once the transactions/orders have been received.
I think that does give me enough confidence to go ahead and prototype a solution. Thank you once again.
I'm busy deploying the local vagrant image and will keep you posted of the findings.
Best regards,
Sanket
________________________________
From: Simon Elliston Ball <si...@simonellistonball.com>
Sent: Monday, March 4, 2019 12:25 PM
To: user@metron.apache.org
Subject: Re: Use case question
Hi Sanket,
This is certainly an interesting case. Metron is deliberately designed for flexibility in terms of ingest and schema, so that non-network data sources and use cases can be accommodated. The one caveat I would suggest is that the Metron pipeline is designed for analytics and detection, but not necessarily for the kind of guaranteed latency you might need for something like a web application experience. While it is streaming and realtime by nature, it can in some circumstances take a second or so to get a message from end to end, particularly if you have a lot of detection or models running, so it's not ideal as part of an interactive process. That said, for the actual detection of fraud, and strange behaviour patterns on your website, it would be a great fit.
Hope that helps,
Simon
On Mon, 4 Mar 2019 at 02:04, Hammad <ha...@gmail.com>> wrote:
Following!!
On Mon, Mar 4, 2019 at 2:29 PM Sanket Sharma <sa...@dukstra.com>> wrote:
Hi,
I've been looking at metron for a few days now and I have a unique use - thought of asking the experts if it makes sense to use metron in this scenario.
My understanding of the project so far is that its a framework built for analyzing cybersecurity threats. This includes analyzing IP packets, network traffics, URLs etc to calculate risk scores etc. The framework also enables data scientists to build and test their models. There are data collection plugins that collect data from variety of sources, stream it over kafka and makes them available for use by various models.
Now, we have a customer facing portal where customers login, submit all kinds of orders and transactions. We were looking at ways to analyze fraud that originates from our portal and I stumbled upon Metron. While we can definitely use Metron for analyzing source traffic, but would it be a good idea to use Metron to analyze the actual transactions themselves? I do understand that we will have to build our models etc. but given that all the heavy lifting is already done, I'm tempted to try Metron for this use case (instead of re-inventing the wheel).
Is this possible/recommended? Or would you recommend using Metron strictly for network related analysis?
Best Regards,
Sanket
--
--
simon elliston ball
@sireb
Re: Use case question
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Hi Sanket,
This is certainly an interesting case. Metron is deliberately designed for
flexibility in terms of ingest and schema, so that non-network data sources
and use cases can be accommodated. The one caveat I would suggest is that
the Metron pipeline is designed for analytics and detection, but not
necessarily for the kind of guaranteed latency you might need for something
like a web application experience. While it is streaming and realtime by
nature, it can in some circumstances take a second or so to get a message
from end to end, particularly if you have a lot of detection or models
running, so it's not ideal as part of an interactive process. That said,
for the actual detection of fraud, and strange behaviour patterns on your
website, it would be a great fit.
Hope that helps,
Simon
On Mon, 4 Mar 2019 at 02:04, Hammad <ha...@gmail.com> wrote:
> Following!!
>
> On Mon, Mar 4, 2019 at 2:29 PM Sanket Sharma <sa...@dukstra.com>
> wrote:
>
>>
>>
>> Hi,
>>
>> I've been looking at metron for a few days now and I have a unique use -
>> thought of asking the experts if it makes sense to use metron in this
>> scenario.
>>
>> My understanding of the project so far is that its a framework built for
>> analyzing cybersecurity threats. This includes analyzing IP packets,
>> network traffics, URLs etc to calculate risk scores etc. The framework also
>> enables data scientists to build and test their models. There are data
>> collection plugins that collect data from variety of sources, stream it
>> over kafka and makes them available for use by various models.
>>
>> Now, we have a customer facing portal where customers login, submit all
>> kinds of orders and transactions. We were looking at ways to analyze fraud
>> that originates from our portal and I stumbled upon Metron. While we can
>> definitely use Metron for analyzing source traffic, but would it be a good
>> idea to use Metron to analyze the actual transactions themselves? I do
>> understand that we will have to build our models etc. but given that all
>> the heavy lifting is already done, I'm tempted to try Metron for this use
>> case (instead of re-inventing the wheel).
>>
>> Is this possible/recommended? Or would you recommend using Metron
>> strictly for network related analysis?
>>
>> Best Regards,
>> Sanket
>>
>
--
--
simon elliston ball
@sireb
Re: Use case question
Posted by Hammad <ha...@gmail.com>.
Following!!
On Mon, Mar 4, 2019 at 2:29 PM Sanket Sharma <sa...@dukstra.com>
wrote:
>
>
> Hi,
>
> I've been looking at metron for a few days now and I have a unique use -
> thought of asking the experts if it makes sense to use metron in this
> scenario.
>
> My understanding of the project so far is that its a framework built for
> analyzing cybersecurity threats. This includes analyzing IP packets,
> network traffics, URLs etc to calculate risk scores etc. The framework also
> enables data scientists to build and test their models. There are data
> collection plugins that collect data from variety of sources, stream it
> over kafka and makes them available for use by various models.
>
> Now, we have a customer facing portal where customers login, submit all
> kinds of orders and transactions. We were looking at ways to analyze fraud
> that originates from our portal and I stumbled upon Metron. While we can
> definitely use Metron for analyzing source traffic, but would it be a good
> idea to use Metron to analyze the actual transactions themselves? I do
> understand that we will have to build our models etc. but given that all
> the heavy lifting is already done, I'm tempted to try Metron for this use
> case (instead of re-inventing the wheel).
>
> Is this possible/recommended? Or would you recommend using Metron strictly
> for network related analysis?
>
> Best Regards,
> Sanket
>