You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by C G <pa...@yahoo.com> on 2007/09/13 16:10:47 UTC
JOIN-type operations with Hadoop...
Consider two row based files. The first has fields:
A B C
the second has fields:
B D E
I want to join these files on the key B, to create records of the form:
A B C D E
So B can be thought of as a primary key, and the second file will only distinct values of B...i.e. no repeats.
I'm trying to reason through how to do this type of join operation in Hadoop but am unsure how to proceed with different "types" of files.
Does the community have any wisdom to share?
Thanks,
C G
---------------------------------
Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
RE: JOIN-type operations with Hadoop...
Posted by Runping Qi <ru...@yahoo-inc.com>.
Check out the data_join package under hadoop/contrib..
It offers a generic framework for doing various joining operations.
Runping
> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Thursday, September 13, 2007 7:11 AM
> To: hadoop-user@lucene.apache.org
> Subject: JOIN-type operations with Hadoop...
>
> Consider two row based files. The first has fields:
>
> A B C
>
> the second has fields:
>
> B D E
>
> I want to join these files on the key B, to create records of the form:
>
> A B C D E
>
> So B can be thought of as a primary key, and the second file will only
> distinct values of B...i.e. no repeats.
>
> I'm trying to reason through how to do this type of join operation in
> Hadoop but am unsure how to proceed with different "types" of files.
>
> Does the community have any wisdom to share?
>
> Thanks,
> C G
>
>
> ---------------------------------
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web
> links.
Re: JOIN-type operations with Hadoop...
Posted by Doug Cutting <cu...@apache.org>.
Ted Dunning wrote:
> Any guess on when this will be?
Now?
http://www.nabble.com/Incubator-Proposal%3A-Pig-p12764836.html
Doug
Re: JOIN-type operations with Hadoop...
Posted by Doug Cutting <cu...@apache.org>.
Ted Dunning wrote:
> Is there any way to add our support to your proposal? Would that even help?
Yes, plese. Join the incubator-general mailing list and participate in
the discussion. Your opinions is welcome there. Only votes from folks
on the Incubator's PMC are binding, but votes from others are useful.
Apache seeks to build diverse developer communities, and if folks who
may become members of the Pig community let themselves be known, then
that provides more evidence that an active community could form around
this technology.
Doug
Re: JOIN-type operations with Hadoop...
Posted by Ted Dunning <td...@veoh.com>.
Is there any way to add our support to your proposal? Would that even help?
On 9/18/07 1:15 PM, "Alan Gates" <ga...@yahoo-inc.com> wrote:
> We've submitted the proposal to Apache. How long it is from here to
> having source available for download I don't know. Hopefully a matter
> of a few weeks.
>
> Alan.
>
> Ted Dunning wrote:
>> Any guess on when this will be?
>>
>>
>> On 9/17/07 12:41 PM, "Alan Gates" <ga...@yahoo-inc.com> wrote:
>>
>>
>>> ... Pig is
>>> in the process of entering the incubator state with Apache, after which
>>> newer versions of pig should be available.
>>>
>>
>>
>
Re: JOIN-type operations with Hadoop...
Posted by Alan Gates <ga...@yahoo-inc.com>.
We've submitted the proposal to Apache. How long it is from here to
having source available for download I don't know. Hopefully a matter
of a few weeks.
Alan.
Ted Dunning wrote:
> Any guess on when this will be?
>
>
> On 9/17/07 12:41 PM, "Alan Gates" <ga...@yahoo-inc.com> wrote:
>
>
>> ... Pig is
>> in the process of entering the incubator state with Apache, after which
>> newer versions of pig should be available.
>>
>
>
Re: JOIN-type operations with Hadoop...
Posted by Ted Dunning <td...@veoh.com>.
Any guess on when this will be?
On 9/17/07 12:41 PM, "Alan Gates" <ga...@yahoo-inc.com> wrote:
> ... Pig is
> in the process of entering the incubator state with Apache, after which
> newer versions of pig should be available.
Re: JOIN-type operations with Hadoop...
Posted by Alan Gates <ga...@yahoo-inc.com>.
Pig is still a 0.x quality product. It works, and generally returns
correct answers to queries. However, at this point error detection and
reporting is limited. The jar available on the Yahoo! research website
is somewhat old and will work with hadoop 0.13.x but not 0.14.x. Pig is
in the process of entering the incubator state with Apache, after which
newer versions of pig should be available.
Alan Gates.
Pig Architect
Yahoo!
Ashish Thusoo wrote:
> Thanks for the pointer.
>
> We did take a look at pig and did find that it some of the constructs
> that we have been talking about. How stable is the pig software? Has
> anyone on this list used it?
>
> Thanks,
> Ashish
>
> -----Original Message-----
> From: Ted Dunning [mailto:tdunning@veoh.com]
> Sent: Thursday, September 13, 2007 11:10 AM
> To: hadoop-user@lucene.apache.org
> Subject: Re: JOIN-type operations with Hadoop...
>
>
>
> See pig.
>
> This one: http://research.yahoo.com/project/pig
>
> Not this one: http://en.wikipedia.org/wiki/Pig
>
> On 9/13/07 10:45 AM, "Ashish Thusoo" <at...@facebook.com> wrote:
>
>
>> On a related note - has anyone seen proposals or ideas for languages
>>
> on
>
>> top of hadoop map/reduce (could even be languages for some sort of
>>
> code
>
>> generators) to make writing the joins easy. It is quite a nightmare to
>> write these joins especially when it involves multiple data sources.
>>
> We
>
>> are thinking of doing something similar. I wanted to find out if
>>
> someone
>
>> else has some ideas to share.
>>
>> Thanks,
>> Ashish
>>
>> -----Original Message-----
>> From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
>> Sent: Thursday, September 13, 2007 7:43 AM
>> To: hadoop-user@lucene.apache.org
>> Subject: RE: JOIN-type operations with Hadoop...
>>
>> We use the directory namespace to distinguish different types of
>>
> files.
>
>> Wrote a simple wrapper around TextInputFormat/SequenceFileInputFormat
>>
> -
>
>> such that they key returned is the pathname (or some component of the
>> pathname). That way u can look at the key - and then decide what kind
>>
> of
>
>> record structure the value encodes and take the proper action.
>>
>> Ping me if u want an example and will be happy to share.
>>
>>
>> -----Original Message-----
>> From: C G [mailto:parallelguy@yahoo.com]
>> Sent: Thursday, September 13, 2007 7:11 AM
>> To: hadoop-user@lucene.apache.org
>> Subject: JOIN-type operations with Hadoop...
>>
>> Consider two row based files. The first has fields:
>>
>> A B C
>>
>> the second has fields:
>>
>> B D E
>>
>> I want to join these files on the key B, to create records of the
>> form:
>>
>> A B C D E
>>
>> So B can be thought of as a primary key, and the second file will
>>
> only
>
>> distinct values of B...i.e. no repeats.
>>
>> I'm trying to reason through how to do this type of join operation
>>
> in
>
>> Hadoop but am unsure how to proceed with different "types" of files.
>>
>> Does the community have any wisdom to share?
>>
>> Thanks,
>> C G
>>
>>
>> ---------------------------------
>> Yahoo! oneSearch: Finally, mobile search that gives answers, not web
>> links.
>>
>
>
Re: JOIN-type operations with Hadoop...
Posted by Ted Dunning <td...@veoh.com>.
I tried to use it, but it is currently tied to the 0.13.1 release of hadoop
as specially patched with the HadoopExe class by the pig group.
I hear that they are working on a new release and also working towards
having a real open source version with nightly builds RSN, but there hasn't
been any externally visible progress lately.
On 9/17/07 11:14 AM, "Ashish Thusoo" <at...@facebook.com> wrote:
> Thanks for the pointer.
>
> We did take a look at pig and did find that it some of the constructs
> that we have been talking about. How stable is the pig software? Has
> anyone on this list used it?
>
> Thanks,
> Ashish
>
> -----Original Message-----
> From: Ted Dunning [mailto:tdunning@veoh.com]
> Sent: Thursday, September 13, 2007 11:10 AM
> To: hadoop-user@lucene.apache.org
> Subject: Re: JOIN-type operations with Hadoop...
>
>
>
> See pig.
>
> This one: http://research.yahoo.com/project/pig
>
> Not this one: http://en.wikipedia.org/wiki/Pig
>
> On 9/13/07 10:45 AM, "Ashish Thusoo" <at...@facebook.com> wrote:
>
>> On a related note - has anyone seen proposals or ideas for languages
> on
>> top of hadoop map/reduce (could even be languages for some sort of
> code
>> generators) to make writing the joins easy. It is quite a nightmare to
>> write these joins especially when it involves multiple data sources.
> We
>> are thinking of doing something similar. I wanted to find out if
> someone
>> else has some ideas to share.
>>
>> Thanks,
>> Ashish
>>
>> -----Original Message-----
>> From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
>> Sent: Thursday, September 13, 2007 7:43 AM
>> To: hadoop-user@lucene.apache.org
>> Subject: RE: JOIN-type operations with Hadoop...
>>
>> We use the directory namespace to distinguish different types of
> files.
>> Wrote a simple wrapper around TextInputFormat/SequenceFileInputFormat
> -
>> such that they key returned is the pathname (or some component of the
>> pathname). That way u can look at the key - and then decide what kind
> of
>> record structure the value encodes and take the proper action.
>>
>> Ping me if u want an example and will be happy to share.
>>
>>
>> -----Original Message-----
>> From: C G [mailto:parallelguy@yahoo.com]
>> Sent: Thursday, September 13, 2007 7:11 AM
>> To: hadoop-user@lucene.apache.org
>> Subject: JOIN-type operations with Hadoop...
>>
>> Consider two row based files. The first has fields:
>>
>> A B C
>>
>> the second has fields:
>>
>> B D E
>>
>> I want to join these files on the key B, to create records of the
>> form:
>>
>> A B C D E
>>
>> So B can be thought of as a primary key, and the second file will
> only
>> distinct values of B...i.e. no repeats.
>>
>> I'm trying to reason through how to do this type of join operation
> in
>> Hadoop but am unsure how to proceed with different "types" of files.
>>
>> Does the community have any wisdom to share?
>>
>> Thanks,
>> C G
>>
>>
>> ---------------------------------
>> Yahoo! oneSearch: Finally, mobile search that gives answers, not web
>> links.
>
RE: JOIN-type operations with Hadoop...
Posted by Ashish Thusoo <at...@facebook.com>.
Thanks for the pointer.
We did take a look at pig and did find that it some of the constructs
that we have been talking about. How stable is the pig software? Has
anyone on this list used it?
Thanks,
Ashish
-----Original Message-----
From: Ted Dunning [mailto:tdunning@veoh.com]
Sent: Thursday, September 13, 2007 11:10 AM
To: hadoop-user@lucene.apache.org
Subject: Re: JOIN-type operations with Hadoop...
See pig.
This one: http://research.yahoo.com/project/pig
Not this one: http://en.wikipedia.org/wiki/Pig
On 9/13/07 10:45 AM, "Ashish Thusoo" <at...@facebook.com> wrote:
> On a related note - has anyone seen proposals or ideas for languages
on
> top of hadoop map/reduce (could even be languages for some sort of
code
> generators) to make writing the joins easy. It is quite a nightmare to
> write these joins especially when it involves multiple data sources.
We
> are thinking of doing something similar. I wanted to find out if
someone
> else has some ideas to share.
>
> Thanks,
> Ashish
>
> -----Original Message-----
> From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
> Sent: Thursday, September 13, 2007 7:43 AM
> To: hadoop-user@lucene.apache.org
> Subject: RE: JOIN-type operations with Hadoop...
>
> We use the directory namespace to distinguish different types of
files.
> Wrote a simple wrapper around TextInputFormat/SequenceFileInputFormat
-
> such that they key returned is the pathname (or some component of the
> pathname). That way u can look at the key - and then decide what kind
of
> record structure the value encodes and take the proper action.
>
> Ping me if u want an example and will be happy to share.
>
>
> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Thursday, September 13, 2007 7:11 AM
> To: hadoop-user@lucene.apache.org
> Subject: JOIN-type operations with Hadoop...
>
> Consider two row based files. The first has fields:
>
> A B C
>
> the second has fields:
>
> B D E
>
> I want to join these files on the key B, to create records of the
> form:
>
> A B C D E
>
> So B can be thought of as a primary key, and the second file will
only
> distinct values of B...i.e. no repeats.
>
> I'm trying to reason through how to do this type of join operation
in
> Hadoop but am unsure how to proceed with different "types" of files.
>
> Does the community have any wisdom to share?
>
> Thanks,
> C G
>
>
> ---------------------------------
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web
> links.
Re: JOIN-type operations with Hadoop...
Posted by Ted Dunning <td...@veoh.com>.
See pig.
This one: http://research.yahoo.com/project/pig
Not this one: http://en.wikipedia.org/wiki/Pig
On 9/13/07 10:45 AM, "Ashish Thusoo" <at...@facebook.com> wrote:
> On a related note - has anyone seen proposals or ideas for languages on
> top of hadoop map/reduce (could even be languages for some sort of code
> generators) to make writing the joins easy. It is quite a nightmare to
> write these joins especially when it involves multiple data sources. We
> are thinking of doing something similar. I wanted to find out if someone
> else has some ideas to share.
>
> Thanks,
> Ashish
>
> -----Original Message-----
> From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
> Sent: Thursday, September 13, 2007 7:43 AM
> To: hadoop-user@lucene.apache.org
> Subject: RE: JOIN-type operations with Hadoop...
>
> We use the directory namespace to distinguish different types of files.
> Wrote a simple wrapper around TextInputFormat/SequenceFileInputFormat -
> such that they key returned is the pathname (or some component of the
> pathname). That way u can look at the key - and then decide what kind of
> record structure the value encodes and take the proper action.
>
> Ping me if u want an example and will be happy to share.
>
>
> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Thursday, September 13, 2007 7:11 AM
> To: hadoop-user@lucene.apache.org
> Subject: JOIN-type operations with Hadoop...
>
> Consider two row based files. The first has fields:
>
> A B C
>
> the second has fields:
>
> B D E
>
> I want to join these files on the key B, to create records of the
> form:
>
> A B C D E
>
> So B can be thought of as a primary key, and the second file will only
> distinct values of B...i.e. no repeats.
>
> I'm trying to reason through how to do this type of join operation in
> Hadoop but am unsure how to proceed with different "types" of files.
>
> Does the community have any wisdom to share?
>
> Thanks,
> C G
>
>
> ---------------------------------
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web
> links.
RE: JOIN-type operations with Hadoop...
Posted by Ashish Thusoo <at...@facebook.com>.
On a related note - has anyone seen proposals or ideas for languages on
top of hadoop map/reduce (could even be languages for some sort of code
generators) to make writing the joins easy. It is quite a nightmare to
write these joins especially when it involves multiple data sources. We
are thinking of doing something similar. I wanted to find out if someone
else has some ideas to share.
Thanks,
Ashish
-----Original Message-----
From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
Sent: Thursday, September 13, 2007 7:43 AM
To: hadoop-user@lucene.apache.org
Subject: RE: JOIN-type operations with Hadoop...
We use the directory namespace to distinguish different types of files.
Wrote a simple wrapper around TextInputFormat/SequenceFileInputFormat -
such that they key returned is the pathname (or some component of the
pathname). That way u can look at the key - and then decide what kind of
record structure the value encodes and take the proper action.
Ping me if u want an example and will be happy to share.
-----Original Message-----
From: C G [mailto:parallelguy@yahoo.com]
Sent: Thursday, September 13, 2007 7:11 AM
To: hadoop-user@lucene.apache.org
Subject: JOIN-type operations with Hadoop...
Consider two row based files. The first has fields:
A B C
the second has fields:
B D E
I want to join these files on the key B, to create records of the
form:
A B C D E
So B can be thought of as a primary key, and the second file will only
distinct values of B...i.e. no repeats.
I'm trying to reason through how to do this type of join operation in
Hadoop but am unsure how to proceed with different "types" of files.
Does the community have any wisdom to share?
Thanks,
C G
---------------------------------
Yahoo! oneSearch: Finally, mobile search that gives answers, not web
links.
RE: JOIN-type operations with Hadoop...
Posted by Joydeep Sen Sarma <js...@facebook.com>.
We use the directory namespace to distinguish different types of files.
Wrote a simple wrapper around TextInputFormat/SequenceFileInputFormat -
such that they key returned is the pathname (or some component of the
pathname). That way u can look at the key - and then decide what kind of
record structure the value encodes and take the proper action.
Ping me if u want an example and will be happy to share.
-----Original Message-----
From: C G [mailto:parallelguy@yahoo.com]
Sent: Thursday, September 13, 2007 7:11 AM
To: hadoop-user@lucene.apache.org
Subject: JOIN-type operations with Hadoop...
Consider two row based files. The first has fields:
A B C
the second has fields:
B D E
I want to join these files on the key B, to create records of the
form:
A B C D E
So B can be thought of as a primary key, and the second file will only
distinct values of B...i.e. no repeats.
I'm trying to reason through how to do this type of join operation in
Hadoop but am unsure how to proceed with different "types" of files.
Does the community have any wisdom to share?
Thanks,
C G
---------------------------------
Yahoo! oneSearch: Finally, mobile search that gives answers, not web
links.