You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Jing Huang (JIRA)" <ji...@apache.org> on 2009/10/16 23:24:31 UTC
[jira] Created: (PIG-1026) [zebra] map split returns null
[zebra] map split returns null
------------------------------
Key: PIG-1026
URL: https://issues.apache.org/jira/browse/PIG-1026
Project: Pig
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Jing Huang
Assignee: Yan Zhou
Fix For: 0.6.0
Here is the test scenario:
final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
//final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
projection: String projection2 = new String("m1#{b}, m2#{x|z}");
User got null pointer exception on reading m1#{b}.
Yan, please refer to the test class:
TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1026) [zebra] map split returns null
Posted by "Chao Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chao Wang updated PIG-1026:
---------------------------
Patch reviewed. +1
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
> Attachments: PIG_1026.patch
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1026) [zebra] map split returns null
Posted by "Jing Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767983#action_12767983 ]
Jing Huang commented on PIG-1026:
---------------------------------
Created a customer scenario with this schema and storage hint: (TestJira1026.java)
final static String STR_SCHEMA = "bcookie:bytes,yuid:bytes, ip:bytes,query_term:bytes,clickinfo:map(String),demog:map(String),page_params:map(String),viewinfo:collection(f1:map(String))";
final static String STR_STORAGE = "[bcookie,yuid,ip,query_term];[clickinfo#{pos|sec|slk|targurl|cost|gpos},page_params#{ipc|vtestid|frcode|pagenum|query}];[clickinfo,page_params,demog];[viewinfo]";
Got NullPointExcepiton.
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
> Attachments: MultipleKeyInMapSplitException.patch
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1026) [zebra] map split returns null
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yan Zhou updated PIG-1026:
--------------------------
Status: Patch Available (was: Open)
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
> Attachments: PIG_1026.patch
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1026) [zebra] map split returns null
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1026:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
All the nightly tests pass. Patch checked in. Thanks Yan!
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
> Attachments: PIG_1026.patch
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1026) [zebra] map split returns null
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yan Zhou updated PIG-1026:
--------------------------
Attachment: PIG_1026.patch
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
> Attachments: PIG_1026.patch
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1026) [zebra] map split returns null
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yan Zhou updated PIG-1026:
--------------------------
Attachment: (was: MultipleKeyInMapSplitException.patch)
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1026) [zebra] map split returns null
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772192#action_12772192 ]
Yan Zhou commented on PIG-1026:
-------------------------------
Another problem is that during STORE, when handleMapSplit method is called to set up the CG schema mapping with the Table schema, an incremented index was used as the column indices to the CG schemas. This works ok if the columns in the CG schemas are in the same order as in the table schema, but will be wrong if the orders are different, causing no values to be stored in some MAP columns in any CG and hence, during LOAD, getValue() returns null.
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
> Attachments: PIG_1026.patch
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1026) [zebra] map split returns null
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774214#action_12774214 ]
Hadoop QA commented on PIG-1026:
--------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423738/PIG_1026.patch
against trunk revision 833266.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/141/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/141/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/141/console
This message is automatically generated.
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
> Attachments: PIG_1026.patch
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1026) [zebra] map split returns null
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1026:
--------------------------------
Comment: was deleted
(was: Created a customer scenario with this schema and storage hint: (TestJira1026.java)
final static String STR_SCHEMA = "bcookie:bytes,yuid:bytes, ip:bytes,query_term:bytes,clickinfo:map(String),demog:map(String),page_params:map(String),viewinfo:collection(f1:map(String))";
final static String STR_STORAGE = "[bcookie,yuid,ip,query_term];[clickinfo#{pos|sec|slk|targurl|cost|gpos},page_params#{ipc|vtestid|frcode|pagenum|query}];[clickinfo,page_params,demog];[viewinfo]";
Got NullPointExcepiton.)
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1026) [zebra] map split returns null
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yan Zhou updated PIG-1026:
--------------------------
Attachment: MultipleKeyInMapSplitException.patch
This patch is not applicable to SVN yet since it's not generated based upon the current SVN image.
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
> Attachments: MultipleKeyInMapSplitException.patch
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1026) [zebra] map split returns null
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767468#action_12767468 ]
Yan Zhou commented on PIG-1026:
-------------------------------
The STR_STORAGE actually has a problem : m2#{x} is actually specified twice. So a syntax exception should have been thrown. I'm going to attach a patch that throws the exception in a proper manner.
After I correct the syntax error, the tests pass, particularly with "[m1#{a|b}, m2#{x|y}]; [m1]; [m2]" as the storage specification.
> [zebra] map split returns null
> ------------------------------
>
> Key: PIG-1026
> URL: https://issues.apache.org/jira/browse/PIG-1026
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Jing Huang
> Assignee: Yan Zhou
> Fix For: 0.6.0
>
>
> Here is the test scenario:
> final static String STR_SCHEMA = "m1:map(string),m2:map(map(int))";
> //final static String STR_STORAGE = "[m1#{a}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1]";
> final static String STR_STORAGE = "[m1#{a}, m2#{x}];[m2#{x|y}]; [m1#{b}, m2#{z}];[m1,m2]";
> projection: String projection2 = new String("m1#{b}, m2#{x|z}");
> User got null pointer exception on reading m1#{b}.
> Yan, please refer to the test class:
> TestNonDefaultWholeMapSplit.java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.