You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by pi song <pi...@gmail.com> on 2008/06/30 17:10:16 UTC
OLAP ?
Has anyone ever looked at OLAP Cube processing on Pig or MapReduce? This
feature should be beneficial for normal businesses (probably not web
scale). I find it pretty simple provided that we can work on relational
abstraction (but might not scale well on really big datasets). I have done a
sketch on how I think it should look like here
http://wiki.apache.org/pig/PigOLAPSketch . Welcome for all
comments/suggestions.
Pi
Re: OLAP ?
Posted by pi song <pi...@gmail.com>.
We will run a combiner in map phase soon right? Potentially I can just take
advantage of that.
On Tue, Jul 1, 2008 at 7:35 AM, Olga Natkovich <ol...@yahoo-inc.com> wrote:
> Pi,
>
> This is a great start!
>
> Looked a little but into proposed implementation - seems like we should
> be able to do a lot of hash-based preaggregation in the map and then
> sort based preaggregation in the combiner.
>
> Olga
>
> > -----Original Message-----
> > From: Chris Olston [mailto:olston@yahoo-inc.com]
> > Sent: Monday, June 30, 2008 8:49 AM
> > To: pig-dev@incubator.apache.org
> > Subject: Re: OLAP ?
> >
> > Awesome idea, Pi!
> >
> > Bottom line: sooner or later somebody is going to package up
> > a "olap- on-mapreduce" solution. Pig seems like a very good
> > vehicle for it.
> >
> > -Chris
> >
> >
> > On Jun 30, 2008, at 8:10 AM, pi song wrote:
> >
> > > Has anyone ever looked at OLAP Cube processing on Pig or
> > MapReduce?
> > > This feature should be beneficial for normal businesses
> > (probably not
> > > web scale). I find it pretty simple provided that we can work on
> > > relational abstraction (but might not scale well on really big
> > > datasets). I have done a sketch on how I think it should look like
> > > here http://wiki.apache.org/pig/PigOLAPSketch . Welcome for all
> > > comments/suggestions.
> > >
> > > Pi
> >
> > --
> > Christopher Olston, Ph.D.
> > Sr. Research Scientist
> > Yahoo! Research
> >
> >
> >
>
RE: OLAP ?
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
Pi,
This is a great start!
Looked a little but into proposed implementation - seems like we should
be able to do a lot of hash-based preaggregation in the map and then
sort based preaggregation in the combiner.
Olga
> -----Original Message-----
> From: Chris Olston [mailto:olston@yahoo-inc.com]
> Sent: Monday, June 30, 2008 8:49 AM
> To: pig-dev@incubator.apache.org
> Subject: Re: OLAP ?
>
> Awesome idea, Pi!
>
> Bottom line: sooner or later somebody is going to package up
> a "olap- on-mapreduce" solution. Pig seems like a very good
> vehicle for it.
>
> -Chris
>
>
> On Jun 30, 2008, at 8:10 AM, pi song wrote:
>
> > Has anyone ever looked at OLAP Cube processing on Pig or
> MapReduce?
> > This feature should be beneficial for normal businesses
> (probably not
> > web scale). I find it pretty simple provided that we can work on
> > relational abstraction (but might not scale well on really big
> > datasets). I have done a sketch on how I think it should look like
> > here http://wiki.apache.org/pig/PigOLAPSketch . Welcome for all
> > comments/suggestions.
> >
> > Pi
>
> --
> Christopher Olston, Ph.D.
> Sr. Research Scientist
> Yahoo! Research
>
>
>
Re: OLAP ?
Posted by Chris Olston <ol...@yahoo-inc.com>.
Awesome idea, Pi!
Bottom line: sooner or later somebody is going to package up a "olap-
on-mapreduce" solution. Pig seems like a very good vehicle for it.
-Chris
On Jun 30, 2008, at 8:10 AM, pi song wrote:
> Has anyone ever looked at OLAP Cube processing on Pig or
> MapReduce? This
> feature should be beneficial for normal businesses (probably not web
> scale). I find it pretty simple provided that we can work on
> relational
> abstraction (but might not scale well on really big datasets). I
> have done a
> sketch on how I think it should look like here
> http://wiki.apache.org/pig/PigOLAPSketch . Welcome for all
> comments/suggestions.
>
> Pi
--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research