You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Vivek Padmanabhan (JIRA)" <ji...@apache.org> on 2012/05/24 15:35:55 UTC
[jira] [Created] (PIG-2721) Wrong output generated while loading
bags as input
Vivek Padmanabhan created PIG-2721:
--------------------------------------
Summary: Wrong output generated while loading bags as input
Key: PIG-2721
URL: https://issues.apache.org/jira/browse/PIG-2721
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.2, 0.9.0
Reporter: Vivek Padmanabhan
{code}
A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
C = order B by id;
dump C;
{code}
The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
The below is the sample input;
{code}
...LKGaHqg-- {(aa,806743)}
..0MI1Y37w-- {(aa,498970)}
..0bnlpJrw-- {(aa,806740)}
..0p0IIhbA-- {(aa,498971),(se,498995)}
..1VkGqvXA-- {(aa,805219)}
{code}
I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
[main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2721:
----------------------------
Fix Version/s: 0.9.3
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2, 0.10.0, 0.11
> Reporter: Vivek Padmanabhan
> Assignee: Koji Noguchi
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: pig-2721-trunk-notestyet.patch, pig-2721-trunk-withtest_v1.patch
>
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2721:
------------------------------
Attachment: pig-2721-trunk-notestyet.patch
Taking the logical plan when used with -t ColumnMapKeyPrune
{noformat}
#-----------------------------------------------
# New Logical Plan:
#-----------------------------------------------
C: (Name: LOStore Schema: id#11:chararray,bttype#14:chararray,cat#15:long)
|
|---B: (Name: LOForEach Schema: id#11:chararray,bttype#14:chararray,cat#15:long)
| |
| (Name: LOGenerate[false,true] Schema: id#11:chararray,bttype#14:chararray,cat#15:long)
| | |
| | id:(Name: Project Type: chararray Uid: 11 Input: 0 Column: (*))
| | |
| | mybag:(Name: Project Type: bag Uid: 12 Input: 1 Column: (*)) <==*HERE1*
| |
| |---(Name: LOInnerLoad[0] Schema: id#11:chararray)
| |
| |---mybag: (Name: LOInnerLoad[1] Schema: bttype#14:chararray,cat#15:long)
|
|---C: (Name: LOSort Schema: id#11:chararray,bttype#14:chararray,cat#15:long) <==*HERE2*
| |
| id:(Name: Project Type: chararray Uid: 11 Input: 0 Column: 0)
|
|---A: (Name: LOForEach Schema: id#11:chararray,mybag#12:bag{#18:tuple(bttype#14:chararray,cat#15:long)})
| |
| (Name: LOGenerate[false,false] Schema: id#11:chararray,mybag#12:bag{#18:tuple(bttype#14:chararray,cat#15:long)})
| | |
| | (Name: Cast Type: chararray Uid: 11)
| | |
| | |---id:(Name: Project Type: bytearray Uid: 11 Input: 0 Column: (*))
| | |
| | (Name: Cast Type: bag Uid: 12)
| | |
| | |---mybag:(Name: Project Type: bytearray Uid: 12 Input: 1 Column: (*))
| |
| |---(Name: LOInnerLoad[0] Schema: id#11:bytearray)
| |
| |---(Name: LOInnerLoad[1] Schema: mybag#12:bytearray)
|
|---A: (Name: LOLoad Schema: id#11:bytearray,mybag#12:bytearray)RequiredFields:null
{noformat}
Tracing the ColumnPrune*, use of 'Uid:12' gets lost at first LOGenerate (*HERE*) part when its projection refers to LOSort (*HERE2*) and checks the schema.
Looking further, LOSort schema was not getting updated by SchemaPatcher when PushDownForEachFlatten swapped ForEach and Sort. This was due to PushDownForEachFlatten.reportChange() not passing the changes it made.
I believe attached patch fixes this issue.
Confirmed logical plan now changes to
{noformat}
$ diff /tmp/before /tmp/after
18c18
< |---C: (Name: LOSort Schema: id#11:chararray,bttype#14:chararray,cat#15:long)
---
> |---C: (Name: LOSort Schema: id#11:chararray,mybag#12:bag{#18:tuple(bttype#14:chararray,cat#15:long)})
{noformat}
and it produces the correct output.
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2
> Reporter: Vivek Padmanabhan
> Attachments: pig-2721-trunk-notestyet.patch
>
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2721:
------------------------------
Attachment: pig-2721-trunk-withtest_v1.patch
Added a test case.
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2, 0.10.0, 0.11
> Reporter: Vivek Padmanabhan
> Attachments: pig-2721-trunk-notestyet.patch, pig-2721-trunk-withtest_v1.patch
>
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430477#comment-13430477 ]
Daniel Dai commented on PIG-2721:
---------------------------------
Committed to 0.9 branch as well as per requested by Rohini.
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2, 0.10.0, 0.11
> Reporter: Vivek Padmanabhan
> Assignee: Koji Noguchi
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: pig-2721-trunk-notestyet.patch, pig-2721-trunk-withtest_v1.patch
>
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283451#comment-13283451 ]
Koji Noguchi commented on PIG-2721:
-----------------------------------
When running the query, it dumps
{noformat}
(...LKGaHqg--)
(..0MI1Y37w--)
(..0bnlpJrw--)
(..0p0IIhbA--)
(..1VkGqvXA--)
{noformat}
but when running with -t ColumnMapKeyPrune or -t PushDownForEachFlatten, it gives the correct output of
{noformat}
(...LKGaHqg--,aa,806743)
(..0MI1Y37w--,aa,498970)
(..0bnlpJrw--,aa,806740)
(..0p0IIhbA--,aa,498971)
(..0p0IIhbA--,se,498995)
(..1VkGqvXA--,aa,805219)
{noformat}
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2
> Reporter: Vivek Padmanabhan
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2721:
----------------------------
Resolution: Fixed
Fix Version/s: 0.10.1
0.11
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
All unit tests pass. Patch committed to 0.10/trunk.
Thanks, Koji!
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2, 0.10.0, 0.11
> Reporter: Vivek Padmanabhan
> Assignee: Koji Noguchi
> Fix For: 0.11, 0.10.1
>
> Attachments: pig-2721-trunk-notestyet.patch, pig-2721-trunk-withtest_v1.patch
>
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi reassigned PIG-2721:
---------------------------------
Assignee: Koji Noguchi
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2, 0.10.0, 0.11
> Reporter: Vivek Padmanabhan
> Assignee: Koji Noguchi
> Attachments: pig-2721-trunk-notestyet.patch, pig-2721-trunk-withtest_v1.patch
>
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2721:
------------------------------
Status: Patch Available (was: Open)
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0, 0.9.2, 0.9.0, 0.11
> Reporter: Vivek Padmanabhan
> Assignee: Koji Noguchi
> Attachments: pig-2721-trunk-notestyet.patch, pig-2721-trunk-withtest_v1.patch
>
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2721:
------------------------------
Affects Version/s: 0.11
0.10.0
This affects 0.9, 0.10, and trunk.
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2, 0.10.0, 0.11
> Reporter: Vivek Padmanabhan
> Attachments: pig-2721-trunk-notestyet.patch, pig-2721-trunk-withtest_v1.patch
>
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2721) Wrong output generated while loading
bags as input
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283758#comment-13283758 ]
Daniel Dai commented on PIG-2721:
---------------------------------
Thanks Koji, you find the right fix. We forget to add changed operator to subPlan in PushDownForEachFlatten. Will commit when tests pass.
> Wrong output generated while loading bags as input
> --------------------------------------------------
>
> Key: PIG-2721
> URL: https://issues.apache.org/jira/browse/PIG-2721
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2, 0.10.0, 0.11
> Reporter: Vivek Padmanabhan
> Attachments: pig-2721-trunk-notestyet.patch, pig-2721-trunk-withtest_v1.patch
>
>
> {code}
> A = LOAD '/user/pvivek/sample' as (id:chararray,mybag:bag{tuple(bttype:chararray,cat:long)});
> B = foreach A generate id,FLATTEN(mybag) AS (bttype, cat);
> C = order B by id;
> dump C;
> {code}
> The above code generates wrong results when executed with Pig 0.10 and Pig 0.9
> The below is the sample input;
> {code}
> ...LKGaHqg-- {(aa,806743)}
> ..0MI1Y37w-- {(aa,498970)}
> ..0bnlpJrw-- {(aa,806740)}
> ..0p0IIhbA-- {(aa,498971),(se,498995)}
> ..1VkGqvXA-- {(aa,805219)}
> {code}
> I think the Pig optimizers are causing this issue.From the logs I can see that the $1 is pruned for the relation A.
> [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for A: $1
> One workaround for this is to disable -t ColumnMapKeyPrune.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira