Prediction within Storm

[Daniela Stoiber <da...@gmail.com>]

Hello

I have the following problem and I hope someone can help me.

I have different devices which are sending me start events which contain a
timestamp and an id to know which device it is. Every device has an
additional file containing 120 values, one value per minute. What I would
like to do is to use these values to predict the next 120 minutes over all
devices in realtime. So anyhow I have to add up all the values of each
minute over all active devices. So to say it is my "predictive model".

So I have to store the events of the devices and to join the values of the
file. But the events have different starting points and I am not sure if I
can realize this within Storm or if i would need any other or additonal
tools.

Thank you in adavance.

Regards,
Daniela

Re: Prediction within Storm

[Daniela Stoiber <da...@gmail.com>]

Hi Enno

Thank you for your reply. My problem is that I have no idea how to deal
with the issue that I receive only a start message from the device but the
values are located in an extra database and that the devices start at
different times.

Each device has a step function with a different length. The step function
consists of minutes and every minute has a different value.

The problem is that I have no idea how to tell the tool to start different
step functions at a different times and how to add up the values of every
minute in realtime. Is that possible within Storm or is it possible to use
a NoSQL database or something like that? Or do I need a tool for predictive
analytics like RapidMinder, R, Orange, Weka and so on.

Unfortunately I have no idea how to solve this problem as I could not find
any examples for such a use case. Most examples for predictive analytics
are based on historical data and I think it even is not a real predictive
analytics use case as I already know the values for the future. Do you have
any examples or an advice how to proceed?

Thank you very much in advance.

Regards,
Daniela

2016-03-07 9:38 GMT+01:00 Enno Shioji <es...@gmail.com>:

> Doesn't sound like there will be any problems but I'm not sure if I'm
> understanding your setup correctly.
>
> What I'd definitely recommend is to implement it in batch first so that
> you can easily test how it performs (in terms of prediction performance).
> This will inform you what kind of state you'll need to keep etc., too.
>
>
>
> > On 7 Mar 2016, at 06:47, Daniela Stoiber <da...@gmail.com>
> wrote:
> >
> > Hello
> >
> > I have the following problem and I hope someone can help me.
> >
> > I have different devices which are sending me start events which contain
> a
> > timestamp and an id to know which device it is. Every device has an
> > additional file containing 120 values, one value per minute. What I would
> > like to do is to use these values to predict the next 120 minutes over
> all
> > devices in realtime. So anyhow I have to add up all the values of each
> > minute over all active devices. So to say it is my "predictive model".
> >
> > So I have to store the events of the devices and to join the values of
> the
> > file. But the events have different starting points and I am not sure if
> I
> > can realize this within Storm or if i would need any other or additonal
> > tools.
> >
> > Thank you in adavance.
> >
> > Regards,
> > Daniela
>

Re: Prediction within Storm

[Enno Shioji <es...@gmail.com>]

Doesn't sound like there will be any problems but I'm not sure if I'm understanding your setup correctly. 

What I'd definitely recommend is to implement it in batch first so that you can easily test how it performs (in terms of prediction performance). This will inform you what kind of state you'll need to keep etc., too.



> On 7 Mar 2016, at 06:47, Daniela Stoiber <da...@gmail.com> wrote:
> 
> Hello
> 
> I have the following problem and I hope someone can help me.
> 
> I have different devices which are sending me start events which contain a
> timestamp and an id to know which device it is. Every device has an
> additional file containing 120 values, one value per minute. What I would
> like to do is to use these values to predict the next 120 minutes over all
> devices in realtime. So anyhow I have to add up all the values of each
> minute over all active devices. So to say it is my "predictive model".
> 
> So I have to store the events of the devices and to join the values of the
> file. But the events have different starting points and I am not sure if I
> can realize this within Storm or if i would need any other or additonal
> tools.
> 
> Thank you in adavance.
> 
> Regards,
> Daniela