You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ashutosh Chauhan (JIRA)" <ji...@apache.org> on 2010/04/03 02:42:27 UTC

[jira] Created: (PIG-1353) Map-side joins

Map-side joins
--------------

                 Key: PIG-1353
                 URL: https://issues.apache.org/jira/browse/PIG-1353
             Project: Pig
          Issue Type: Improvement
          Components: impl
            Reporter: Ashutosh Chauhan
            Assignee: Ashutosh Chauhan


Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 

Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1353) Map-side outer joins

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich updated PIG-1353:
--------------------------------

    Summary: Map-side outer joins  (was: Map-side joins)

> Map-side outer joins
> --------------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.8.0
>
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1353) Map-side joins

Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ashutosh Chauhan updated PIG-1353:
----------------------------------

    Attachment: pig-1353.patch

An illustrative patch which achieves this.

> Map-side joins
> --------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>         Attachments: pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1353) Map-side joins

Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ashutosh Chauhan updated PIG-1353:
----------------------------------

          Status: Resolved  (was: Patch Available)
    Release Note: 
With this patch, it is now possible to perform [left|right|full] outer joins on two tables as well as inner joins on more then two tables in Pig in map-side if data is sorted and one of the loader implements {{CollectableLoader}} interface. Primary algorithm is based on sort-merge join. 

Additional implementation details:
1) No other operations can be done between load and join statements.
2) Data must be sorted in ASC order.
3) Nulls are considered smaller then everything. So, if data contains null keys, they should occur before anything else.
4) Left-most loader must implement CollectableLoader interface as well as OrderedLoadFunc.
5) All other loaders must implement IndexableLoadFunc.   

Note that Zebra loader satisfies all of these conditions, so can be used out of box.
Similiar conditions apply to map-side cogroups (PIG-1309) as well.  
      Resolution: Fixed

Patch checked-in.


> Map-side joins
> --------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.8.0
>
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (PIG-1353) Map-side joins

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857866#action_12857866 ] 

Alan Gates commented on PIG-1353:
---------------------------------

Why did you convert all of the visitor methods in LogToPhysTranslator and LOVisitor from public to protected?  That seems unrelated to your changes.



> Map-side joins
> --------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.8.0
>
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (PIG-1353) Map-side joins

Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857149#action_12857149 ] 

Ashutosh Chauhan commented on PIG-1353:
---------------------------------------

Hudson.. Oh Hudson.. when y'll get better ! Ran the full test suite. All of them passed. Ran test-patch:
{noformat}
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 12 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

{noformat}

Patch is ready for review.

> Map-side joins
> --------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.8.0
>
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (PIG-1353) Map-side joins

Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ashutosh Chauhan updated PIG-1353:
----------------------------------

    Attachment: pig-1353.patch

Running through hudson. 

> Map-side joins
> --------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (PIG-1353) Map-side joins

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857896#action_12857896 ] 

Alan Gates commented on PIG-1353:
---------------------------------

Ok, I'm +1 on the patch.

> Map-side joins
> --------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.8.0
>
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (PIG-1353) Map-side outer joins

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich updated PIG-1353:
--------------------------------

    Release Note: 
With this patch, it is now possible to perform [left|right|full] outer joins on two tables as well as inner joins on more then two tables in Pig in map-side if data is sorted and one of the loader implements {{CollectableLoader}} interface. Primary algorithm is based on sort-merge join. 

Additional implementation details:
1) No other operations can be done between load and join statements.
2) Data must be sorted in ASC order.
3) Nulls are considered smaller then everything. So, if data contains null keys, they should occur before anything else.
4) Left-most loader must implement CollectableLoader interface as well as OrderedLoadFunc.
5) All other loaders must implement IndexableLoadFunc.   

Note that Zebra loader satisfies all of these conditions, so can be used out of box.
Similiar conditions apply to map-side cogroups (PIG-1309) as well.  

Example:
A = load 'data1' using org.apache.hadoop.zebra.pig.TableLoader('', 'sorted');
B = load 'data2' using org.apache.hadoop.zebra.pig.TableLoader('', 'sorted');
C = join A by id left, B by id using 'merge';
.....

  was:
With this patch, it is now possible to perform [left|right|full] outer joins on two tables as well as inner joins on more then two tables in Pig in map-side if data is sorted and one of the loader implements {{CollectableLoader}} interface. Primary algorithm is based on sort-merge join. 

Additional implementation details:
1) No other operations can be done between load and join statements.
2) Data must be sorted in ASC order.
3) Nulls are considered smaller then everything. So, if data contains null keys, they should occur before anything else.
4) Left-most loader must implement CollectableLoader interface as well as OrderedLoadFunc.
5) All other loaders must implement IndexableLoadFunc.   

Note that Zebra loader satisfies all of these conditions, so can be used out of box.
Similiar conditions apply to map-side cogroups (PIG-1309) as well.  


> Map-side outer joins
> --------------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.8.0
>
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1353) Map-side joins

Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857887#action_12857887 ] 

Ashutosh Chauhan commented on PIG-1353:
---------------------------------------

Ya, visibility change of visitor methods from public to protected is unrelated to the issue. I did it as a part of cleanup of LogToPhyTranslator. I dont see a reason why visitor methods should be public. in general we strive to not make things public and all the usage of those methods were in same package, so changing them from public to protected is a safe choice. 

> Map-side joins
> --------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.8.0
>
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (PIG-1353) Map-side joins

Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ashutosh Chauhan updated PIG-1353:
----------------------------------

           Status: Patch Available  (was: Open)
    Fix Version/s: 0.8.0

> Map-side joins
> --------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.8.0
>
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (PIG-1353) Map-side outer joins

Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ashutosh Chauhan updated PIG-1353:
----------------------------------

    Release Note: 
With this patch, it is now possible to perform [left|right|full] outer joins on two tables as well as inner joins on more then two tables in Pig in map-side if data is sorted and loaders implement required interfaces. Primary algorithm is based on sort-merge join. 

Following preconditions should be met in order to use this feature:
1) No other operations can be done between load and join statements.
2) Data must be sorted on join keys in ASC order.
3) Nulls are considered smaller then everything. So, if data contains null keys, they should occur before anything else.
4) Left-most loader must implement {CollectableLoader} interface as well as {OrderedLoadFunc}.
5) All other loaders must implement {IndexableLoadFunc}.   
6) Type information must be provided in schema for all the loaders. 

Note that Zebra loader satisfies all of these conditions, so can be used out of box.

Similar conditions apply to map-side cogroups (PIG-1309) as well.  

Example:
A = load 'data1' using org.apache.hadoop.zebra.pig.TableLoader('id:int', 'sorted');
B = load 'data2' using org.apache.hadoop.zebra.pig.TableLoader('id:int', 'sorted');
C = join A by id left, B by id using 'merge';
.....

  was:
With this patch, it is now possible to perform [left|right|full] outer joins on two tables as well as inner joins on more then two tables in Pig in map-side if data is sorted and one of the loader implements {{CollectableLoader}} interface. Primary algorithm is based on sort-merge join. 

Additional implementation details:
1) No other operations can be done between load and join statements.
2) Data must be sorted in ASC order.
3) Nulls are considered smaller then everything. So, if data contains null keys, they should occur before anything else.
4) Left-most loader must implement CollectableLoader interface as well as OrderedLoadFunc.
5) All other loaders must implement IndexableLoadFunc.   

Note that Zebra loader satisfies all of these conditions, so can be used out of box.
Similiar conditions apply to map-side cogroups (PIG-1309) as well.  

Example:
A = load 'data1' using org.apache.hadoop.zebra.pig.TableLoader('', 'sorted');
B = load 'data2' using org.apache.hadoop.zebra.pig.TableLoader('', 'sorted');
C = join A by id left, B by id using 'merge';
.....


> Map-side outer joins
> --------------------
>
>                 Key: PIG-1353
>                 URL: https://issues.apache.org/jira/browse/PIG-1353
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.8.0
>
>         Attachments: pig-1353.patch, pig-1353.patch
>
>
> Pig already has couple of map-side join implementations: Merge Join and Fragmented-Replicate Join. But both of them are pretty restrictive. Merge Join can only join two tables and that too can only do inner join. FR Join can join multiple relations, but it can also only do inner and left outer joins. Further it restricts the sizes of side relations. It will be nice if we can do map side joins on multiple tables as well do inner, left outer, right outer and full outer joins. 
> Lot of groundwork for this has already been done in PIG-1309. Remaining will be tracked in this jira.   

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.