You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Zhiwei Lin <zh...@gmail.com> on 2012/05/21 22:01:33 UTC
Stream data processing
I have large volume of stream log data. Each data record contains a time
stamp, which is very important to the analysis.
For example, I have data format like this:
(1) 20:30:21 01/April/2012 AAAAA.............
(2) 20:30:51 01/April/2012 BBBB.............
(3) 21:30:21 01/April/2012 bbbb.............
Moreover, new data comes every few minutes.
I have to calculate the probability of the occurrence "bbbb" given the
occurrence of "BBBB" (where BBBB occurs earlier than bbbb). So, it is
really time-dependant.
I wonder if Hadoop is the right platform for this job? Is there any
package available for this kind of work?
Thank you.
Zhiwei
Re: Stream data processing
Posted by Zhiwei Lin <zh...@gmail.com>.
Hi Bobby,
Thank you. Great help.
Zhiwei
On 22 May 2012 14:52, Robert Evans <ev...@yahoo-inc.com> wrote:
> If you want the results to come out instantly Map/Reduce is not the proper
> choice. Map/Reduce is designed for batch processing. It can do small
> batches, but the overhead of launching the map/redcue jobs can be very high
> compared to the amount of processing you are doing. I personally would
> look into using either Storm, S4, or some other realtime stream processing
> framework. From what you have said it sounds like you probably want to use
> Storm, as it can be used to guarantee that each event is processed once and
> only once. You can also store your results into HDFS if you want, perhaps
> through HBASE, if you need to do further processing on the data.
>
> --Bobby Evans
>
> On 5/22/12 5:02 AM, "Zhiwei Lin" <zh...@gmail.com> wrote:
>
> Hi Robert,
> Thank you.
> How quickly do you have to get the result out once the new data is added?
> If possible, I hope to get the result instantly.
>
> How far back in time do you have to look for BBBB from the occurrence of
> bbbb?
> The time slot is not constant. It depends on the "last" occurrence of BBBB
> in front of bbbb. So, I need to look up the history to get the last BBBB
> in this case.
>
> Do you have to do this for all combinations of values or is it just a small
> subset of values?
> I think this depends on the time of last occurrence of BBBB in the history.
> If BBBB rarely occurred, then the early stage data has to be taken into
> account.
>
> Definitely, I think HDFS is a good place to store the data I have (the size
> of daily log is above 1GB). But I am not sure if Map/Reduce can help to
> handle the stated problem.
>
> Zhiwei
>
>
> On 21 May 2012 22:07, Robert Evans <ev...@yahoo-inc.com> wrote:
>
> > Zhiwei,
> >
> > How quickly do you have to get the result out once the new data is added?
> > How far back in time do you have to look for BBBB from the occurrence of
> > bbbb? Do you have to do this for all combinations of values or is it
> just
> > a small subset of values?
> >
> > --Bobby Evans
> >
> > On 5/21/12 3:01 PM, "Zhiwei Lin" <zh...@gmail.com> wrote:
> >
> > I have large volume of stream log data. Each data record contains a time
> > stamp, which is very important to the analysis.
> > For example, I have data format like this:
> > (1) 20:30:21 01/April/2012 AAAAA.............
> > (2) 20:30:51 01/April/2012 BBBB.............
> > (3) 21:30:21 01/April/2012 bbbb.............
> >
> > Moreover, new data comes every few minutes.
> > I have to calculate the probability of the occurrence "bbbb" given the
> > occurrence of "BBBB" (where BBBB occurs earlier than bbbb). So, it is
> > really time-dependant.
> >
> > I wonder if Hadoop is the right platform for this job? Is there any
> > package available for this kind of work?
> >
> > Thank you.
> >
> > Zhiwei
> >
> >
>
>
> --
>
> Best wishes.
>
> Zhiwei
>
>
--
Best wishes.
Zhiwei
Re: Stream data processing
Posted by Robert Evans <ev...@yahoo-inc.com>.
If you want the results to come out instantly Map/Reduce is not the proper choice. Map/Reduce is designed for batch processing. It can do small batches, but the overhead of launching the map/redcue jobs can be very high compared to the amount of processing you are doing. I personally would look into using either Storm, S4, or some other realtime stream processing framework. From what you have said it sounds like you probably want to use Storm, as it can be used to guarantee that each event is processed once and only once. You can also store your results into HDFS if you want, perhaps through HBASE, if you need to do further processing on the data.
--Bobby Evans
On 5/22/12 5:02 AM, "Zhiwei Lin" <zh...@gmail.com> wrote:
Hi Robert,
Thank you.
How quickly do you have to get the result out once the new data is added?
If possible, I hope to get the result instantly.
How far back in time do you have to look for BBBB from the occurrence of
bbbb?
The time slot is not constant. It depends on the "last" occurrence of BBBB
in front of bbbb. So, I need to look up the history to get the last BBBB
in this case.
Do you have to do this for all combinations of values or is it just a small
subset of values?
I think this depends on the time of last occurrence of BBBB in the history.
If BBBB rarely occurred, then the early stage data has to be taken into
account.
Definitely, I think HDFS is a good place to store the data I have (the size
of daily log is above 1GB). But I am not sure if Map/Reduce can help to
handle the stated problem.
Zhiwei
On 21 May 2012 22:07, Robert Evans <ev...@yahoo-inc.com> wrote:
> Zhiwei,
>
> How quickly do you have to get the result out once the new data is added?
> How far back in time do you have to look for BBBB from the occurrence of
> bbbb? Do you have to do this for all combinations of values or is it just
> a small subset of values?
>
> --Bobby Evans
>
> On 5/21/12 3:01 PM, "Zhiwei Lin" <zh...@gmail.com> wrote:
>
> I have large volume of stream log data. Each data record contains a time
> stamp, which is very important to the analysis.
> For example, I have data format like this:
> (1) 20:30:21 01/April/2012 AAAAA.............
> (2) 20:30:51 01/April/2012 BBBB.............
> (3) 21:30:21 01/April/2012 bbbb.............
>
> Moreover, new data comes every few minutes.
> I have to calculate the probability of the occurrence "bbbb" given the
> occurrence of "BBBB" (where BBBB occurs earlier than bbbb). So, it is
> really time-dependant.
>
> I wonder if Hadoop is the right platform for this job? Is there any
> package available for this kind of work?
>
> Thank you.
>
> Zhiwei
>
>
--
Best wishes.
Zhiwei
Re: Stream data processing
Posted by Zhiwei Lin <zh...@gmail.com>.
Hi Robert,
Thank you.
How quickly do you have to get the result out once the new data is added?
If possible, I hope to get the result instantly.
How far back in time do you have to look for BBBB from the occurrence of
bbbb?
The time slot is not constant. It depends on the "last" occurrence of BBBB
in front of bbbb. So, I need to look up the history to get the last BBBB
in this case.
Do you have to do this for all combinations of values or is it just a small
subset of values?
I think this depends on the time of last occurrence of BBBB in the history.
If BBBB rarely occurred, then the early stage data has to be taken into
account.
Definitely, I think HDFS is a good place to store the data I have (the size
of daily log is above 1GB). But I am not sure if Map/Reduce can help to
handle the stated problem.
Zhiwei
On 21 May 2012 22:07, Robert Evans <ev...@yahoo-inc.com> wrote:
> Zhiwei,
>
> How quickly do you have to get the result out once the new data is added?
> How far back in time do you have to look for BBBB from the occurrence of
> bbbb? Do you have to do this for all combinations of values or is it just
> a small subset of values?
>
> --Bobby Evans
>
> On 5/21/12 3:01 PM, "Zhiwei Lin" <zh...@gmail.com> wrote:
>
> I have large volume of stream log data. Each data record contains a time
> stamp, which is very important to the analysis.
> For example, I have data format like this:
> (1) 20:30:21 01/April/2012 AAAAA.............
> (2) 20:30:51 01/April/2012 BBBB.............
> (3) 21:30:21 01/April/2012 bbbb.............
>
> Moreover, new data comes every few minutes.
> I have to calculate the probability of the occurrence "bbbb" given the
> occurrence of "BBBB" (where BBBB occurs earlier than bbbb). So, it is
> really time-dependant.
>
> I wonder if Hadoop is the right platform for this job? Is there any
> package available for this kind of work?
>
> Thank you.
>
> Zhiwei
>
>
--
Best wishes.
Zhiwei
Re: Stream data processing
Posted by Robert Evans <ev...@yahoo-inc.com>.
Zhiwei,
How quickly do you have to get the result out once the new data is added? How far back in time do you have to look for BBBB from the occurrence of bbbb? Do you have to do this for all combinations of values or is it just a small subset of values?
--Bobby Evans
On 5/21/12 3:01 PM, "Zhiwei Lin" <zh...@gmail.com> wrote:
I have large volume of stream log data. Each data record contains a time
stamp, which is very important to the analysis.
For example, I have data format like this:
(1) 20:30:21 01/April/2012 AAAAA.............
(2) 20:30:51 01/April/2012 BBBB.............
(3) 21:30:21 01/April/2012 bbbb.............
Moreover, new data comes every few minutes.
I have to calculate the probability of the occurrence "bbbb" given the
occurrence of "BBBB" (where BBBB occurs earlier than bbbb). So, it is
really time-dependant.
I wonder if Hadoop is the right platform for this job? Is there any
package available for this kind of work?
Thank you.
Zhiwei