You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Yuanyuan Tian <yt...@us.ibm.com> on 2009/05/28 20:17:08 UTC
Need help to understand joins in Pig
Hi,
I am a new Pig user. I need help to understand joins in Pig. The following
are my questions about how joins are implemented in Pig:
1. I know that a normal join is translated to a co-group operation. Does
the implementation of co-group have the same flavor as the join algorithm
in the Hadoop contrib: use map to extract join keys and partition both
input tables, then use reduce to do the actual join?
And if so, does co-group spill to disk to avoid running out of RAM?
2. There is also a fragment replicate join in Pig for joining a big table
with a small table. Does each mapper/reducer read the small table directly
from HDFS, or does it first copy the table to local file system and read
from local file system?
And if there are multiple mappers/reducers per node, does a new
mapper/reducer always have to reread the small table even the previous
mappers/reducers have already done so?
Thanks very much,
YY
Re: Need help to understand joins in Pig
Posted by Yuanyuan Tian <yt...@us.ibm.com>.
Thanks, Alan. Your answers are very helpful.
YY
Alan Gates
<gates@yahoo-inc.
com> To
pig-user@hadoop.apache.org
05/28/2009 03:14 cc
PM
Subject
Re: Need help to understand joins
Please respond to in Pig
pig-user@hadoop.a
pache.org
On May 28, 2009, at 11:17 AM, Yuanyuan Tian wrote:
>
>
> Hi,
>
> I am a new Pig user. I need help to understand joins in Pig. The
> following
> are my questions about how joins are implemented in Pig:
>
> 1. I know that a normal join is translated to a co-group operation.
> Does
> the implementation of co-group have the same flavor as the join
> algorithm
> in the Hadoop contrib: use map to extract join keys and partition both
> input tables, then use reduce to do the actual join?
Yes, each map annotates the tuples with information on which input it
is from. We then set the partitioner to partition just on the join
key, and have the reducer do a secondary sort on join key plus input
number. The first input (or in an n-way join, the 1st through n-1th
inputs) are read in the reducer and cached in memory, while the last
input is streamed through and the join done. Pig does detect when it
is running low on memory and spill the cached rows to disk. We have
had a number of issues with this code and are looking at ways to
improve it (see http://wiki.apache.org/pig/PigMemory).
>
>
> And if so, does co-group spill to disk to avoid running out of RAM?
>
> 2. There is also a fragment replicate join in Pig for joining a big
> table
> with a small table. Does each mapper/reducer read the small table
> directly
> from HDFS, or does it first copy the table to local file system and
> read
> from local file system?
>
> And if there are multiple mappers/reducers per node, does a new
> mapper/reducer always have to reread the small table even the previous
> mappers/reducers have already done so?
At this point it reads the small table from HDFS and does not take
advantage of opportunities to share the file with maps on the same
machine.
Alan.
Re: Need help to understand joins in Pig
Posted by Alan Gates <ga...@yahoo-inc.com>.
On May 28, 2009, at 11:17 AM, Yuanyuan Tian wrote:
>
>
> Hi,
>
> I am a new Pig user. I need help to understand joins in Pig. The
> following
> are my questions about how joins are implemented in Pig:
>
> 1. I know that a normal join is translated to a co-group operation.
> Does
> the implementation of co-group have the same flavor as the join
> algorithm
> in the Hadoop contrib: use map to extract join keys and partition both
> input tables, then use reduce to do the actual join?
Yes, each map annotates the tuples with information on which input it
is from. We then set the partitioner to partition just on the join
key, and have the reducer do a secondary sort on join key plus input
number. The first input (or in an n-way join, the 1st through n-1th
inputs) are read in the reducer and cached in memory, while the last
input is streamed through and the join done. Pig does detect when it
is running low on memory and spill the cached rows to disk. We have
had a number of issues with this code and are looking at ways to
improve it (see http://wiki.apache.org/pig/PigMemory).
>
>
> And if so, does co-group spill to disk to avoid running out of RAM?
>
> 2. There is also a fragment replicate join in Pig for joining a big
> table
> with a small table. Does each mapper/reducer read the small table
> directly
> from HDFS, or does it first copy the table to local file system and
> read
> from local file system?
>
> And if there are multiple mappers/reducers per node, does a new
> mapper/reducer always have to reread the small table even the previous
> mappers/reducers have already done so?
At this point it reads the small table from HDFS and does not take
advantage of opportunities to share the file with maps on the same
machine.
Alan.