You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Harry Potter <ha...@yahoo.com> on 2012/01/12 13:51:16 UTC
MR algorithms?
How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
Re: MR algorithms?
Posted by Ioan Eugen Stan <st...@gmail.com>.
2012/1/12 Harry Potter <ha...@yahoo.com>:
> How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
You should also have a look at anomaly detection. It similar to
classification only more effective when when the data is skewed (more
good packets then bad ones).
There is a good introduction to the subject by Andrew Ng on http://ml-class.org.
Hope this helps,
--
Ioan Eugen Stan
http://ieugen.blogspot.com/
Re: MR algorithms?
Posted by Ted Dunning <te...@gmail.com>.
Aside from the real-time requirements, you need to do feature extraction
from your packets and decide what "similar" means for packets. Once you
have that, then you are good to go.
For real time handling, you need long lived processes. The recommendation
of Storm and S4 are useful, but not sufficient since you need a framework
that will allow general scheduling and resource allocation. There are two
paths to take with that. First, Mesos is an Apache project that provides a
resource allocator over a cluster that has been in production at Twitter
for some time. Secondly, Yarn is the resource allocator component of the
upcoming 0.23 release of Hadoop. I say upcoming because the releases so
far are pretty much just alpha maturity.
On Thu, Jan 12, 2012 at 8:01 AM, Harry Potter <ha...@yahoo.com>wrote:
> yes we have to analyse realtime network packets for intrusion prevention.
> What technologies other than MR can be used? We implemented simple kmeans
> clustering using map reduce. how can we extend that for clustering network
> packets also?
>
>
> ________________________________
> From: Sebastian Schelter <ss...@apache.org>
> To: user@mahout.apache.org
> Sent: Thursday, January 12, 2012 6:24 PM
> Subject: Re: MR algorithms?
>
> Hi Harry Potter (nice name, btw),
>
> what do you mean with "live"? Do you want to analyze a stream of network
> packets in realtime? MapReduce aims at offline batch processing so you
> should better use another technology stack.
>
> --sebastian
>
> On 12.01.2012 13:51, Harry Potter wrote:
> > How can we make use of map-reduce algorithm for performing
> classification/clustering of live network packets?
>
Re: MR algorithms?
Posted by Mathias Herberts <ma...@gmail.com>.
> yes we have to analyse realtime network packets for intrusion prevention. What technologies other than MR can be used? We implemented simple kmeans clustering using map reduce. how can we extend that for clustering network packets also?
You should have a look at storm
(https://github.com/nathanmarz/storm/wiki) or S4 (http://s4.io/)
>
>
> ________________________________
> From: Sebastian Schelter <ss...@apache.org>
> To: user@mahout.apache.org
> Sent: Thursday, January 12, 2012 6:24 PM
> Subject: Re: MR algorithms?
>
> Hi Harry Potter (nice name, btw),
>
> what do you mean with "live"? Do you want to analyze a stream of network
> packets in realtime? MapReduce aims at offline batch processing so you
> should better use another technology stack.
>
> --sebastian
>
> On 12.01.2012 13:51, Harry Potter wrote:
>> How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
Re: MR algorithms?
Posted by Harry Potter <ha...@yahoo.com>.
yes we have to analyse realtime network packets for intrusion prevention. What technologies other than MR can be used? We implemented simple kmeans clustering using map reduce. how can we extend that for clustering network packets also?
________________________________
From: Sebastian Schelter <ss...@apache.org>
To: user@mahout.apache.org
Sent: Thursday, January 12, 2012 6:24 PM
Subject: Re: MR algorithms?
Hi Harry Potter (nice name, btw),
what do you mean with "live"? Do you want to analyze a stream of network
packets in realtime? MapReduce aims at offline batch processing so you
should better use another technology stack.
--sebastian
On 12.01.2012 13:51, Harry Potter wrote:
> How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
Re: MR algorithms?
Posted by Sebastian Schelter <ss...@apache.org>.
Hi Harry Potter (nice name, btw),
what do you mean with "live"? Do you want to analyze a stream of network
packets in realtime? MapReduce aims at offline batch processing so you
should better use another technology stack.
--sebastian
On 12.01.2012 13:51, Harry Potter wrote:
> How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
Re: MR algorithms?
Posted by Kate Ericson <er...@cs.colostate.edu>.
I know it's a bit late, but to throw in my two cents, Granules
(http://granules.cs.colostate.edu/) is designed to work with streaming
data (such as incoming packets), and supports the mapreduce paradigm,
as well as arbitrary processing schemes.
We've done some work with classifying EEG signals in real-time for BCI
applications, it may be a good fit for classifying network packets.
-Kate
On Fri, Jan 13, 2012 at 11:21 AM, Josh Patterson <jo...@cloudera.com> wrote:
> If you want to wander off into the more esoteric applications of rogue
> packet detection you could look at an older paper of mine from my
> master's:
>
> TinyTermite
>
> http://www.mendeley.com/research/tinytermite-secure-routing-algorithm-intel-mote-2-sensor-network-platform-4/
>
> In this case I added a technique called "suspicion" to an existing
> routing algorithm where I used ant colony task-response-thresholds
> along with a "task" signal (here, would be whatever you found to be
> 'suspicious' yet not overly outright "bad") which allowed the system
> to learn patterns as well as respond with appropriate responses based
> on the severity/type of the attack.
>
> An example of the offline analysis with online detection would be: you
> could learn what are "somewhat suspicious" activities by examining the
> data in hadoop with MR, and then use those as
> "task-response-thresholds" in your networking setup.
>
> (I've gone off the page a bit from the MR world, but hey --- when you
> get the chance to bring up swarm algorithms, gotta jump at the chance,
> right? =) )
>
> JP
>
> On Fri, Jan 13, 2012 at 2:19 AM, Dawid Weiss
> <da...@cs.put.poznan.pl> wrote:
>> Can't remember the reference from the top off my head right now, but I
>> remember talking to a person who did packet analysis and the approach
>> they took was to extract similarity features to known hostile packets/
>> packet streams using suffix arrays/ suffix trees (this most likely
>> requires offline construction in spite of O(n) theoretical cost but
>> will be very fast at runtime). Then the problem becomes a kNN
>> classifier, although you can probably use other classifiers and mix in
>> other features as well.
>>
>> Dawid
>>
>> On Fri, Jan 13, 2012 at 4:11 AM, Lance Norskog <go...@gmail.com> wrote:
>>> Classification is a "sample v.s. model" exercise, while clustering is
>>> "many samples against each other". Clustering is more natively batch
>>> than classification. So, this does sound like a classification
>>> problem.
>>>
>>> Maybe another way to explain it. Classification (and recommendation)
>>> machine learning algorithms create and then use a "data model", a
>>> statistical fingerprint of your training data. They then apply the
>>> model against test samples.
>>>
>>> You would, as Josh said, build a corpus of your training data which is
>>> already classified (called 'labeling'). Some of the Mahout
>>> implementations support taking one sample at a time and comparing it
>>> to the model: SGD, for example. An "online" algorithm takes one sample
>>> at a time, does an analysis is "roughly" constant time and returns.
>>>
>>> In other algorithms, the data model is far too large to fit in memory,
>>> and so they do piecewise calculations with many samples partly
>>> processed at the same time.
>>>
>>> On Thu, Jan 12, 2012 at 7:52 AM, Josh Patterson <jo...@cloudera.com> wrote:
>>>> From the perspective of packets as time series, I used to collect the
>>>> "time series" in a hadoop cluster and study the data to find patterns.
>>>> Once you have a model that you can apply in MR, in certain cases that
>>>> model could be used in streaming data collectors as a "real time
>>>> classifier" to flag sequences of time series / packets. We did
>>>> something similar to this with the openPDC project:
>>>>
>>>> http://openpdc.codeplex.com
>>>>
>>>> JP
>>>>
>>>> On Thu, Jan 12, 2012 at 7:51 AM, Harry Potter <ha...@yahoo.com> wrote:
>>>>> How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
>>>>
>>>>
>>>>
>>>> --
>>>> Twitter: @jpatanooga
>>>> Solution Architect @ Cloudera
>>>> hadoop: http://www.cloudera.com
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> goksron@gmail.com
>>>
>
>
>
> --
> Twitter: @jpatanooga
> Solution Architect @ Cloudera
> hadoop: http://www.cloudera.com
Re: MR algorithms?
Posted by Josh Patterson <jo...@cloudera.com>.
If you want to wander off into the more esoteric applications of rogue
packet detection you could look at an older paper of mine from my
master's:
TinyTermite
http://www.mendeley.com/research/tinytermite-secure-routing-algorithm-intel-mote-2-sensor-network-platform-4/
In this case I added a technique called "suspicion" to an existing
routing algorithm where I used ant colony task-response-thresholds
along with a "task" signal (here, would be whatever you found to be
'suspicious' yet not overly outright "bad") which allowed the system
to learn patterns as well as respond with appropriate responses based
on the severity/type of the attack.
An example of the offline analysis with online detection would be: you
could learn what are "somewhat suspicious" activities by examining the
data in hadoop with MR, and then use those as
"task-response-thresholds" in your networking setup.
(I've gone off the page a bit from the MR world, but hey --- when you
get the chance to bring up swarm algorithms, gotta jump at the chance,
right? =) )
JP
On Fri, Jan 13, 2012 at 2:19 AM, Dawid Weiss
<da...@cs.put.poznan.pl> wrote:
> Can't remember the reference from the top off my head right now, but I
> remember talking to a person who did packet analysis and the approach
> they took was to extract similarity features to known hostile packets/
> packet streams using suffix arrays/ suffix trees (this most likely
> requires offline construction in spite of O(n) theoretical cost but
> will be very fast at runtime). Then the problem becomes a kNN
> classifier, although you can probably use other classifiers and mix in
> other features as well.
>
> Dawid
>
> On Fri, Jan 13, 2012 at 4:11 AM, Lance Norskog <go...@gmail.com> wrote:
>> Classification is a "sample v.s. model" exercise, while clustering is
>> "many samples against each other". Clustering is more natively batch
>> than classification. So, this does sound like a classification
>> problem.
>>
>> Maybe another way to explain it. Classification (and recommendation)
>> machine learning algorithms create and then use a "data model", a
>> statistical fingerprint of your training data. They then apply the
>> model against test samples.
>>
>> You would, as Josh said, build a corpus of your training data which is
>> already classified (called 'labeling'). Some of the Mahout
>> implementations support taking one sample at a time and comparing it
>> to the model: SGD, for example. An "online" algorithm takes one sample
>> at a time, does an analysis is "roughly" constant time and returns.
>>
>> In other algorithms, the data model is far too large to fit in memory,
>> and so they do piecewise calculations with many samples partly
>> processed at the same time.
>>
>> On Thu, Jan 12, 2012 at 7:52 AM, Josh Patterson <jo...@cloudera.com> wrote:
>>> From the perspective of packets as time series, I used to collect the
>>> "time series" in a hadoop cluster and study the data to find patterns.
>>> Once you have a model that you can apply in MR, in certain cases that
>>> model could be used in streaming data collectors as a "real time
>>> classifier" to flag sequences of time series / packets. We did
>>> something similar to this with the openPDC project:
>>>
>>> http://openpdc.codeplex.com
>>>
>>> JP
>>>
>>> On Thu, Jan 12, 2012 at 7:51 AM, Harry Potter <ha...@yahoo.com> wrote:
>>>> How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
>>>
>>>
>>>
>>> --
>>> Twitter: @jpatanooga
>>> Solution Architect @ Cloudera
>>> hadoop: http://www.cloudera.com
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
--
Twitter: @jpatanooga
Solution Architect @ Cloudera
hadoop: http://www.cloudera.com
Re: MR algorithms?
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
Can't remember the reference from the top off my head right now, but I
remember talking to a person who did packet analysis and the approach
they took was to extract similarity features to known hostile packets/
packet streams using suffix arrays/ suffix trees (this most likely
requires offline construction in spite of O(n) theoretical cost but
will be very fast at runtime). Then the problem becomes a kNN
classifier, although you can probably use other classifiers and mix in
other features as well.
Dawid
On Fri, Jan 13, 2012 at 4:11 AM, Lance Norskog <go...@gmail.com> wrote:
> Classification is a "sample v.s. model" exercise, while clustering is
> "many samples against each other". Clustering is more natively batch
> than classification. So, this does sound like a classification
> problem.
>
> Maybe another way to explain it. Classification (and recommendation)
> machine learning algorithms create and then use a "data model", a
> statistical fingerprint of your training data. They then apply the
> model against test samples.
>
> You would, as Josh said, build a corpus of your training data which is
> already classified (called 'labeling'). Some of the Mahout
> implementations support taking one sample at a time and comparing it
> to the model: SGD, for example. An "online" algorithm takes one sample
> at a time, does an analysis is "roughly" constant time and returns.
>
> In other algorithms, the data model is far too large to fit in memory,
> and so they do piecewise calculations with many samples partly
> processed at the same time.
>
> On Thu, Jan 12, 2012 at 7:52 AM, Josh Patterson <jo...@cloudera.com> wrote:
>> From the perspective of packets as time series, I used to collect the
>> "time series" in a hadoop cluster and study the data to find patterns.
>> Once you have a model that you can apply in MR, in certain cases that
>> model could be used in streaming data collectors as a "real time
>> classifier" to flag sequences of time series / packets. We did
>> something similar to this with the openPDC project:
>>
>> http://openpdc.codeplex.com
>>
>> JP
>>
>> On Thu, Jan 12, 2012 at 7:51 AM, Harry Potter <ha...@yahoo.com> wrote:
>>> How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
>>
>>
>>
>> --
>> Twitter: @jpatanooga
>> Solution Architect @ Cloudera
>> hadoop: http://www.cloudera.com
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: MR algorithms?
Posted by Lance Norskog <go...@gmail.com>.
Classification is a "sample v.s. model" exercise, while clustering is
"many samples against each other". Clustering is more natively batch
than classification. So, this does sound like a classification
problem.
Maybe another way to explain it. Classification (and recommendation)
machine learning algorithms create and then use a "data model", a
statistical fingerprint of your training data. They then apply the
model against test samples.
You would, as Josh said, build a corpus of your training data which is
already classified (called 'labeling'). Some of the Mahout
implementations support taking one sample at a time and comparing it
to the model: SGD, for example. An "online" algorithm takes one sample
at a time, does an analysis is "roughly" constant time and returns.
In other algorithms, the data model is far too large to fit in memory,
and so they do piecewise calculations with many samples partly
processed at the same time.
On Thu, Jan 12, 2012 at 7:52 AM, Josh Patterson <jo...@cloudera.com> wrote:
> From the perspective of packets as time series, I used to collect the
> "time series" in a hadoop cluster and study the data to find patterns.
> Once you have a model that you can apply in MR, in certain cases that
> model could be used in streaming data collectors as a "real time
> classifier" to flag sequences of time series / packets. We did
> something similar to this with the openPDC project:
>
> http://openpdc.codeplex.com
>
> JP
>
> On Thu, Jan 12, 2012 at 7:51 AM, Harry Potter <ha...@yahoo.com> wrote:
>> How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
>
>
>
> --
> Twitter: @jpatanooga
> Solution Architect @ Cloudera
> hadoop: http://www.cloudera.com
--
Lance Norskog
goksron@gmail.com
Re: MR algorithms?
Posted by Josh Patterson <jo...@cloudera.com>.
>From the perspective of packets as time series, I used to collect the
"time series" in a hadoop cluster and study the data to find patterns.
Once you have a model that you can apply in MR, in certain cases that
model could be used in streaming data collectors as a "real time
classifier" to flag sequences of time series / packets. We did
something similar to this with the openPDC project:
http://openpdc.codeplex.com
JP
On Thu, Jan 12, 2012 at 7:51 AM, Harry Potter <ha...@yahoo.com> wrote:
> How can we make use of map-reduce algorithm for performing classification/clustering of live network packets?
--
Twitter: @jpatanooga
Solution Architect @ Cloudera
hadoop: http://www.cloudera.com