You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Simone Gianni <si...@apache.org> on 2008/02/17 23:03:18 UTC
DSP Filters
Hi all,
are DSP filters (FIR,IIR etc..) something useful for this project?
Digital Signal Processing is used in some fields of Machine Learning,
mainly to pre-process or to filter data before ingestion in Machine
Learning algorithms.
Simone
--
Simone Gianni
http://simonegianni.it/
Re: DSP Filters
Posted by Ted Dunning <td...@veoh.com>.
I think that DSP filters are very important for some kinds of algorithms.
And it sounds like you have something already working.
That sounds very encouraging.
On 2/18/08 7:28 AM, "Simone Gianni" <si...@apache.org> wrote:
> Hi Ted,
> thanks for your answer.
>
> I don't yet have a clear idea of the scope of this project, so I was
> just asking if DSP filters are something in the scope of the project. I
> have an already developed library of DSP filters, so if you think they
> are useful or will be useful or will fall into the scope of the project,
> I can donate them.
>
> I have used DSP filters for statistical analysis of long time series
> (like logs) a couple of years ago, but not for preparation of data for
> ingestion into a ML algorithm, simply to isolate and notify to the user
> when something strange happened, like an uncommon frequency of traffic
> from a certain ip.
>
> So, as you correctly pointed out, I have not yet felt the pain of
> combining DSP with other ML algorithms. If you feel like it could give
> some benefit, code is there ready to be donated.
>
> Simone
>
> Ted Dunning wrote:
>> To my mind, the key questions in general here are whether the problem needs
>> parallel processing at scale and whether map/reduce can be profitably
>> applied to the problem.
>>
>> FIR and IIR input filters for time-series are definitely important in some
>> domains and for extremely large time series or large filters, these may
>> actually benefit from map-reduce implementation.
>>
>> So the next question is whether you would have benefited by having these
>> operations operate in parallel. I definitely can say that all of the time
>> series problems I have had to deal with had very moderate scale and thus
>> didn't really need any subsequent processing. You sound like you have had a
>> different experience. If so, then please do implement these since we would
>> all learn from your experience.
>>
>> On the other hand, implementing something just because somebody might
>> someday find it useful would not be very helpful. The biggest reason for
>> this is that if you don't feel the pain already, you probably can't guess
>> what pains somebody else will feel when they need this capability.
>>
>> On 2/17/08 2:03 PM, "Simone Gianni" <si...@apache.org> wrote:
>>
>>
>>> Hi all,
>>> are DSP filters (FIR,IIR etc..) something useful for this project?
>>> Digital Signal Processing is used in some fields of Machine Learning,
>>> mainly to pre-process or to filter data before ingestion in Machine
>>> Learning algorithms.
>>>
>>> Simone
>>>
>>
>>
>
Re: DSP Filters
Posted by Simone Gianni <si...@apache.org>.
Hi Ted,
thanks for your answer.
I don't yet have a clear idea of the scope of this project, so I was
just asking if DSP filters are something in the scope of the project. I
have an already developed library of DSP filters, so if you think they
are useful or will be useful or will fall into the scope of the project,
I can donate them.
I have used DSP filters for statistical analysis of long time series
(like logs) a couple of years ago, but not for preparation of data for
ingestion into a ML algorithm, simply to isolate and notify to the user
when something strange happened, like an uncommon frequency of traffic
from a certain ip.
So, as you correctly pointed out, I have not yet felt the pain of
combining DSP with other ML algorithms. If you feel like it could give
some benefit, code is there ready to be donated.
Simone
Ted Dunning wrote:
> To my mind, the key questions in general here are whether the problem needs
> parallel processing at scale and whether map/reduce can be profitably
> applied to the problem.
>
> FIR and IIR input filters for time-series are definitely important in some
> domains and for extremely large time series or large filters, these may
> actually benefit from map-reduce implementation.
>
> So the next question is whether you would have benefited by having these
> operations operate in parallel. I definitely can say that all of the time
> series problems I have had to deal with had very moderate scale and thus
> didn't really need any subsequent processing. You sound like you have had a
> different experience. If so, then please do implement these since we would
> all learn from your experience.
>
> On the other hand, implementing something just because somebody might
> someday find it useful would not be very helpful. The biggest reason for
> this is that if you don't feel the pain already, you probably can't guess
> what pains somebody else will feel when they need this capability.
>
> On 2/17/08 2:03 PM, "Simone Gianni" <si...@apache.org> wrote:
>
>
>> Hi all,
>> are DSP filters (FIR,IIR etc..) something useful for this project?
>> Digital Signal Processing is used in some fields of Machine Learning,
>> mainly to pre-process or to filter data before ingestion in Machine
>> Learning algorithms.
>>
>> Simone
>>
>
>
Re: DSP Filters
Posted by Ted Dunning <td...@veoh.com>.
To my mind, the key questions in general here are whether the problem needs
parallel processing at scale and whether map/reduce can be profitably
applied to the problem.
FIR and IIR input filters for time-series are definitely important in some
domains and for extremely large time series or large filters, these may
actually benefit from map-reduce implementation.
So the next question is whether you would have benefited by having these
operations operate in parallel. I definitely can say that all of the time
series problems I have had to deal with had very moderate scale and thus
didn't really need any subsequent processing. You sound like you have had a
different experience. If so, then please do implement these since we would
all learn from your experience.
On the other hand, implementing something just because somebody might
someday find it useful would not be very helpful. The biggest reason for
this is that if you don't feel the pain already, you probably can't guess
what pains somebody else will feel when they need this capability.
On 2/17/08 2:03 PM, "Simone Gianni" <si...@apache.org> wrote:
> Hi all,
> are DSP filters (FIR,IIR etc..) something useful for this project?
> Digital Signal Processing is used in some fields of Machine Learning,
> mainly to pre-process or to filter data before ingestion in Machine
> Learning algorithms.
>
> Simone