You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2012/12/17 17:11:40 UTC
[jira] [Commented] (PIG-3051) java.lang.IndexOutOfBoundsException
failure with LimitOptimizer + ColumnPruning
[ https://issues.apache.org/jira/browse/PIG-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534028#comment-13534028 ]
Rohini Palaniswamy commented on PIG-3051:
-----------------------------------------
Resetting the attached LOSort operator of the ProjectExpression to the newSort is good. But found an issue with the copy not setting the label, type and Uid.
{code}
@Test
public void testPIG3051() throws Exception {
String[] input = {
"1,2,3,4",
"2,3,4,1",
"3,4,1,2",
"4,1,2,3"
};
Util.createLocalInputFile( "a.txt", input);
String query = "A =load 'a.txt' using PigStorage(',') as (a1:chararray, a2:chararray, a3:chararray, a4:chararray);" +
"B = foreach A generate a2,a3,a4;" +
"G = order B by a4;" +
"U1 = limit G 3;" +
"U2 = foreach U1 generate a4;" +
"store G into 'g' using PigStorage();" +
"store U2 into 'u2' using PigStorage(); ";
try {
PigServer pigServer = new PigServer(ExecType.LOCAL);
pigServer.registerQuery(query);
} catch (Exception e) {
e.printStackTrace();
}
}
{code}
sort.mSortColPlans - a4:(Name: Project Type: chararray Uid: 4 Input: 0 Column: 2)
newSort.mSortColPlans - (Name: Project Type: null Uid: null Input: 0 Column: 2)
{code}
{code}
> java.lang.IndexOutOfBoundsException failure with LimitOptimizer + ColumnPruning
> --------------------------------------------------------------------------------
>
> Key: PIG-3051
> URL: https://issues.apache.org/jira/browse/PIG-3051
> Project: Pig
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.10.0, 0.11
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Fix For: 0.11
>
> Attachments: pig-3051-v1.1-withe2etest.txt, pig-3051-v1-withouttest.txt
>
>
> Had a user hitting
> "Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1" error when he had multiple stores and limit in his code.
> I couldn't reproduce this with short pig code (due to ColumnPruning somehow not happening when shortened), but here's a snippet.
> {noformat}
> ...
> G3 = FOREACH G2 GENERATE sortCol, FLATTEN(group) as label, (long)COUNT(G1) as cnt;
> G4 = ORDER G3 BY cnt DESC PARALLEL 25;
> ONEROW = LIMIT G4 1;
> U1 = FOREACH ONEROW GENERATE 3 as sortcol, 'somelabel' as label, cnt;
> store U1 into 'u1' using PigStorage();
> store G4 into 'g4' using PigStorage();
> {noformat}
> With '-t ColumnMapKeyPrune', job didn't hit the error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira