You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Doug Cutting <cu...@apache.org> on 2007/04/26 22:10:30 UTC
Pig | Yahoo! Research
FYI
http://research.yahoo.com/project/pig
Doug
Re: Pig | Yahoo! Research
Posted by "Peter W." <pe...@marketingbrokers.com>.
Hi,
For the casual developer without academic
parallel computing theory experience to draw
from Hadoop's benefits are apparent but hard
to implement. Design patterns?
Pig's use of tuples, bags and data atoms
are more intuitive for newbies and feel like
MapReduce object return types.
Click-stream or web graph data can be
opened and distributively filtered over
namenodes by those with SQL familiarity.
JSON-type formatting means you can also
leverage any Javascript web presentation
frameworks with the output.
Brilliant Yahoo!.
Peter W.
Re: Pig | Yahoo! Research
Posted by Michael Plax <mi...@mcycorp.com>.
I also found a post -Yahoo Pig and Google Sawzall (Greg Linden ) about it
http://glinden.blogspot.com/
Michael
----- Original Message -----
From: "Doug Cutting" <cu...@apache.org>
To: <ha...@lucene.apache.org>
Sent: Thursday, April 26, 2007 1:10 PM
Subject: Pig | Yahoo! Research
> FYI
>
> http://research.yahoo.com/project/pig
>
> Doug
>
Re: Pig | Yahoo! Research
Posted by Chris <ch...@gmail.com>.
1.
Google's big Table is for store Data with Big size.
File System file size limited for the data in storing.
The computer programmer can get and store data easily in the Big
Table architecture.
2. Pig in Yahoo using Query Language to get the Result. The cluster
computing like MapReduce working . It will let response time shortly
with Map/Reduce producing and computing in the cluster-liking machine.
So , The project is not the same. But I feel Pig in Yahoo is like
MapReduce in Google.
If I get fault, plz tell me. Thank you.
==============================
Chris Lin
http://search20.portal20.com.tw
chrislin@pchome.com.tw
Taipei , Taiwan.
------------------------------------------------------------
2007/4/27, Jim Kellerman <ji...@powerset.com>:
> Can someone comment on how Pig compares with Bigtable?
Re: Pig | Yahoo! Research
Posted by Benjamin Reed <br...@yahoo-inc.com>.
I haven't been reading this list like I should...
Pig is meant to provide a more powerful and simple abstraction for writing
distributed processing logic than mapreduce by itself provides.
Power:
- We support joins.
- We provide the ability to select fields from records that will be passed to
a function so that general functions can be written and reused.
- We do some amount of optimization at the moment (more in the future) to
reduce the number of actual jobs that get run.
Simplicity:
- The model has one kind of function: an eval function. It processes one
record at a time. A dataset can have records grouped together or sorted or
filtered or have a projection applied to it, but functions just need to work
on one record at a time. If a=load 'dataset', MAP is foreach a generate
map(*) and REDUCE is b=group a by $0; foreach b generate reduce(*)
- Since Pig Latin is a simple language that can be used directly. We have a
simple interpreter called GRUNT that users can interact with to submit jobs.
- Eventually we would like to embed Pig Latin into Perl, Python and Ruby to
create Erlpay, Ythonpay and Ubyray, but we are a bit low on developer
bandwidth. We believe that by embedding Pig Latin into existing languages we
would end up with a much more powerful, well know, and natural environment to
work in as opposed to creating our own language like Sawzall.
ben
On Thursday 26 April 2007 15:17:24 Ian Holsman wrote:
> Jim Kellerman wrote:
> > Can someone comment on how Pig compares with Bigtable?
> >
> > On Thu, 2007-04-26 at 13:10 -0700, Doug Cutting wrote:
> >> FYI
> >>
> >> http://research.yahoo.com/project/pig
> >>
> >> Doug
>
> my understanding is
>
> bigtable/hbase stores the data
> mapreduce/hadoop manipulates/creates the data to be stored in bigtable
> via functions, and controls the distribution
> sawzall/pig is a query language to extract information from it. I think
> it would use create functions for mapreduce/hadoop to run.
>
> regards
> Ian
Re: Pig | Yahoo! Research
Posted by Andrew Hitchcock <ad...@gmail.com>.
My understanding is that both Pig and Sawzall are intended for batch
data processing and provide an abstraction layer above MapReduce.
BigTable, on the other hand, is more of a long-term random-access database.
On 4/26/07, Ian Holsman <li...@holsman.net> wrote:
> Jim Kellerman wrote:
> > Can someone comment on how Pig compares with Bigtable?
> >
> > On Thu, 2007-04-26 at 13:10 -0700, Doug Cutting wrote:
> >
> >> FYI
> >>
> >> http://research.yahoo.com/project/pig
> >>
> >> Doug
> >>
> my understanding is
>
> bigtable/hbase stores the data
> mapreduce/hadoop manipulates/creates the data to be stored in bigtable
> via functions, and controls the distribution
> sawzall/pig is a query language to extract information from it. I think
> it would use create functions for mapreduce/hadoop to run.
>
> regards
> Ian
>
>
Re: Pig | Yahoo! Research
Posted by Ian Holsman <li...@holsman.net>.
Jim Kellerman wrote:
> Can someone comment on how Pig compares with Bigtable?
>
> On Thu, 2007-04-26 at 13:10 -0700, Doug Cutting wrote:
>
>> FYI
>>
>> http://research.yahoo.com/project/pig
>>
>> Doug
>>
my understanding is
bigtable/hbase stores the data
mapreduce/hadoop manipulates/creates the data to be stored in bigtable
via functions, and controls the distribution
sawzall/pig is a query language to extract information from it. I think
it would use create functions for mapreduce/hadoop to run.
regards
Ian
Re: Pig | Yahoo! Research
Posted by Jim Kellerman <ji...@powerset.com>.
Can someone comment on how Pig compares with Bigtable?
On Thu, 2007-04-26 at 13:10 -0700, Doug Cutting wrote:
> FYI
>
> http://research.yahoo.com/project/pig
>
> Doug
--
Jim Kellerman, Senior Engineer; Powerset
jim@powerset.com