You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ying He (JIRA)" <ji...@apache.org> on 2009/12/09 02:54:18 UTC
[jira] Created: (PIG-1135) skewed join partitioner returns negative
partition index
skewed join partitioner returns negative partition index
---------------------------------------------------------
Key: PIG-1135
URL: https://issues.apache.org/jira/browse/PIG-1135
Project: Pig
Issue Type: Improvement
Reporter: Ying He
Assignee: Sriranjan Manjunath
Fragmented replicated join has a few limitations:
- One of the tables needs to be loaded into memory
- Join is limited to two tables
Skewed join partitions the table and joins the records in the reduce phase. It computes a histogram of the key space to account for skewing in the input records. Further, it adjusts the number of reducers depending on the key distribution.
We need to implement the skewed join in pig.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1135) skewed join partitioner returns negative
partition index
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1135:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
patch committed to both the trunk and the branch. Thanks, Ying!
> skewed join partitioner returns negative partition index
> ---------------------------------------------------------
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
> Issue Type: Improvement
> Reporter: Ying He
> Assignee: Sriranjan Manjunath
> Fix For: 0.6.0
>
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1135) skewed join partitioner returns
negative partition index
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788014#action_12788014 ]
Hadoop QA commented on PIG-1135:
--------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427422/PIG_1135.patch
against trunk revision 888704.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/107/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/107/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/107/console
This message is automatically generated.
> skewed join partitioner returns negative partition index
> ---------------------------------------------------------
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
> Issue Type: Improvement
> Reporter: Ying He
> Assignee: Sriranjan Manjunath
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1135) skewed join partitioner returns negative
partition index
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ying He updated PIG-1135:
-------------------------
Description: skewed join returns negative reducer index (was: Fragmented replicated join has a few limitations:
- One of the tables needs to be loaded into memory
- Join is limited to two tables
Skewed join partitions the table and joins the records in the reduce phase. It computes a histogram of the key space to account for skewing in the input records. Further, it adjusts the number of reducers depending on the key distribution.
We need to implement the skewed join in pig.)
> skewed join partitioner returns negative partition index
> ---------------------------------------------------------
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
> Issue Type: Improvement
> Reporter: Ying He
> Assignee: Sriranjan Manjunath
>
> skewed join returns negative reducer index
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1135) skewed join partitioner returns
negative partition index
Posted by "Sriranjan Manjunath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787912#action_12787912 ]
Sriranjan Manjunath commented on PIG-1135:
------------------------------------------
Ran skewed join end-end / unit tests against this patch and random sample loader, and they passed.
> skewed join partitioner returns negative partition index
> ---------------------------------------------------------
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
> Issue Type: Improvement
> Reporter: Ying He
> Assignee: Sriranjan Manjunath
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1135) skewed join partitioner returns negative
partition index
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1135:
--------------------------------
Fix Version/s: 0.6.0
> skewed join partitioner returns negative partition index
> ---------------------------------------------------------
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
> Issue Type: Improvement
> Reporter: Ying He
> Assignee: Sriranjan Manjunath
> Fix For: 0.6.0
>
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1135) skewed join partitioner returns
negative partition index
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788141#action_12788141 ]
Olga Natkovich commented on PIG-1135:
-------------------------------------
+1, changes look good
> skewed join partitioner returns negative partition index
> ---------------------------------------------------------
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
> Issue Type: Improvement
> Reporter: Ying He
> Assignee: Sriranjan Manjunath
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1135) skewed join partitioner returns negative
partition index
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ying He updated PIG-1135:
-------------------------
Status: Patch Available (was: Open)
> skewed join partitioner returns negative partition index
> ---------------------------------------------------------
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
> Issue Type: Improvement
> Reporter: Ying He
> Assignee: Sriranjan Manjunath
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1135) skewed join partitioner returns negative
partition index
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ying He updated PIG-1135:
-------------------------
Attachment: PIG_1135.patch
if reducer index is greater than 128, the index of streaming table becomes negative because byte is used as data type.
It is fixed by changing partition index from byte to integer.
> skewed join partitioner returns negative partition index
> ---------------------------------------------------------
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
> Issue Type: Improvement
> Reporter: Ying He
> Assignee: Sriranjan Manjunath
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.