Re: Re: Evaluating the ML for netflow, proxy, dns

[Christos Mathas <ma...@gmail.com>]

Hi,

I'm currently evaluating spot and I'm also using ROC-AUC, but I'm not 
experienced in these evaluation methods. If you could share some of the 
data you mention or set up a meeting as you say to explain how you're 
conducting the evaluation process it would be really useful.

Also, I would like to ask about the topics parameter:

How can I calculate an approximation of the number of topics based on 
the setup/traffic of my network?

Thanks in advance


On 06/05/2017 05:40 PM, Lujan Moreno, Gustavo wrote:
> Hi,
>
> I have plenty of information about how to compute spot-ml performance 
> in terms of ROC-AUC and we are getting good results with internal 
> data. In order to compute the AUC we are injecting artificial attacks 
> and track their rank on the final score. Once we know the ranks of the 
> attacks we can compute the AUC. On proxy we are getting > 0.90 on AUC 
> and on net flow >0.99. I understand improvements on DNS are being done 
> and results of AUC should be computed soon.
>
> The data is internal and I don’t think I can share it. The attacks are 
> being generated using breaking point. I will ask if I can share this.
>
> We can also set up a meeting to explain how I’m conducting the 
> evaluation process.
>
> Best,
>
> Gustavo
>
>
>
> From: Giacomo Bernardi <mino@minux.it <ma...@minux.it>>
> Reply-To: "user@spot.incubator.apache.org 
> <ma...@spot.incubator.apache.org>" 
> <user@spot.incubator.apache.org <ma...@spot.incubator.apache.org>>
> Date: Saturday, June 3, 2017 at 1:28 PM
> To: "user@spot.incubator.apache.org 
> <ma...@spot.incubator.apache.org>" 
> <user@spot.incubator.apache.org <ma...@spot.incubator.apache.org>>
> Subject: Re: Evaluating the ML for netflow, proxy, dns
>
> +1
> On the same boat.
>
> I've been trying Spot on two different networks (a large office, and a 
> large event venue) but I haven't been able to extract useful anomalies 
> so far. I understand the algorithm, but to be honest all the results I 
> get from spot-ml->spot-oa seem a bit random and not "anomalous" at all.
>
> I played a bit with the LDA parameters but that hasn't helped.
>
> Thanks.
> ps: i'm only looking at the netflow feed, no dns/proxy.
>
>
> On 3 June 2017 at 17:48, D.Anil <anilcseiitm2006@gmail.com 
> <ma...@gmail.com>> wrote:
>
>     Hi,
>
>     I would like to know if any of us have test data (for netflow,
>     proxy, dns) that have certain events to be detected as
>     threat/anomalies such that we can run ML on top of the test data
>     to evaluate the ML algorithm.
>
>     I would like to understand the concept of threshold in the score
>     calculation of LDA algorithm and how can we fine tune the
>     threshold to give best results for the model.
>
>     Looking forward to hear the experiences from the community.
>
>     Thanks,
>     Anil.
>
>