You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "David Ciemiewicz (JIRA)" <ji...@apache.org> on 2009/03/10 04:18:51 UTC
[jira] Created: (PIG-710) Filtering bag in nested foreach does not
produce expected results
Filtering bag in nested foreach does not produce expected results
-----------------------------------------------------------------
Key: PIG-710
URL: https://issues.apache.org/jira/browse/PIG-710
Project: Pig
Issue Type: Bug
Reporter: David Ciemiewicz
I have an idiom I used to use in older versions of pig (prior to types branch) which would group into a collection and then filter the output if any of the collection contained a particular string.
This relies on FILTER statements within a FOREACH ... { ... GENERATE ... } statement.
ORDER ... BY in the FOREACH ... { ... GENERATE ... } statement does not seem to have a problem so it seems to be something isolated to the FILTER.
{code}
A = load 'filterbug.data' using PigStorage() as ( id, str );
B = group A by ( id );
describe B;
dump B;
D = foreach B generate
group,
COUNT(A),
A.str;
describe D;
dump D;
C = foreach B {
D = order A by str;
matchedcount = COUNT(D);
generate
group,
matchedcount as matchedcount,
D.str;
};
describe C;
dump C;
Cfiltered = foreach B {
D = filter A by (
str matches 'hello'
);
matchedcount = COUNT(D);
generate
group,
matchedcount as matchedcount,
A.str;
};
describe Cfiltered;
dump Cfiltered;
{code}
Here's the output:
{code}
-bash-3.00$ pig -exectype local -latest filterbug.pig
USING: /grid/0/gs/pig/current
B: {group: bytearray,A: {id: bytearray,str: bytearray}}
2009-03-10 03:14:14,838 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
2009-03-10 03:14:14,839 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
(a,{(a,hello),(a,goodbye)})
(b,{(b,goodbye)})
(c,{(c,hello),(c,hello),(c,hello)})
(d,{(d,what)})
D: {group: bytearray,long,str: {str: bytearray}}
2009-03-10 03:14:14,920 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
2009-03-10 03:14:14,920 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
(a,2L,{(hello),(goodbye)})
(b,1L,{(goodbye)})
(c,3L,{(hello),(hello),(hello)})
(d,1L,{(what)})
C: {group: bytearray,matchedcount: long,str: {str: bytearray}}
2009-03-10 03:14:14,985 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
2009-03-10 03:14:14,985 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
(a,2L,{(goodbye),(hello)})
(b,1L,{(goodbye)})
(c,3L,{(hello),(hello),(hello)})
(d,1L,{(what)})
2009-03-10 03:14:15,018 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 1 time(s).
Cfiltered: {group: bytearray,matchedcount: long,str: {str: bytearray}}
2009-03-10 03:14:15,044 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 1 time(s).
2009-03-10 03:14:15,057 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
2009-03-10 03:14:15,057 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
(a,1L,{(hello),(goodbye)})
{code}
What I expect for the output of Cfiltered is actually:
(a,1L,{(hello),(goodbye)})
(b,0L,{(goodbye)})
(c,3L,{(hello),(hello),(hello)})
(d,0L,{(what)})
The data file is:
{code}
a hello
a goodbye
b goodbye
c hello
c hello
c hello
d what
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-710) Filtering bag in nested foreach does not
produce expected results
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath resolved PIG-710.
--------------------------------
Resolution: Duplicate
Fix Version/s: 0.3.0
Assignee: Pradeep Kamath
This issue has the same root cause as PIG-514 - hence marking this duplicate - the fix for this issue will also be tracked in PIG-514
> Filtering bag in nested foreach does not produce expected results
> -----------------------------------------------------------------
>
> Key: PIG-710
> URL: https://issues.apache.org/jira/browse/PIG-710
> Project: Pig
> Issue Type: Bug
> Reporter: David Ciemiewicz
> Assignee: Pradeep Kamath
> Fix For: 0.3.0
>
>
> I have an idiom I used to use in older versions of pig (prior to types branch) which would group into a collection and then filter the output if any of the collection contained a particular string.
> This relies on FILTER statements within a FOREACH ... { ... GENERATE ... } statement.
> ORDER ... BY in the FOREACH ... { ... GENERATE ... } statement does not seem to have a problem so it seems to be something isolated to the FILTER.
> {code}
> A = load 'filterbug.data' using PigStorage() as ( id, str );
> B = group A by ( id );
> describe B;
> dump B;
> D = foreach B generate
> group,
> COUNT(A),
> A.str;
> describe D;
> dump D;
> C = foreach B {
> D = order A by str;
> matchedcount = COUNT(D);
> generate
> group,
> matchedcount as matchedcount,
> D.str;
> };
> describe C;
> dump C;
> Cfiltered = foreach B {
> D = filter A by (
> str matches 'hello'
> );
> matchedcount = COUNT(D);
> generate
> group,
> matchedcount as matchedcount,
> A.str;
> };
> describe Cfiltered;
> dump Cfiltered;
> {code}
> Here's the output:
> {code}
> -bash-3.00$ pig -exectype local -latest filterbug.pig
> USING: /grid/0/gs/pig/current
> B: {group: bytearray,A: {id: bytearray,str: bytearray}}
> 2009-03-10 03:14:14,838 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-03-10 03:14:14,839 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> (a,{(a,hello),(a,goodbye)})
> (b,{(b,goodbye)})
> (c,{(c,hello),(c,hello),(c,hello)})
> (d,{(d,what)})
> D: {group: bytearray,long,str: {str: bytearray}}
> 2009-03-10 03:14:14,920 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-03-10 03:14:14,920 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> (a,2L,{(hello),(goodbye)})
> (b,1L,{(goodbye)})
> (c,3L,{(hello),(hello),(hello)})
> (d,1L,{(what)})
> C: {group: bytearray,matchedcount: long,str: {str: bytearray}}
> 2009-03-10 03:14:14,985 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-03-10 03:14:14,985 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> (a,2L,{(goodbye),(hello)})
> (b,1L,{(goodbye)})
> (c,3L,{(hello),(hello),(hello)})
> (d,1L,{(what)})
> 2009-03-10 03:14:15,018 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 1 time(s).
> Cfiltered: {group: bytearray,matchedcount: long,str: {str: bytearray}}
> 2009-03-10 03:14:15,044 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 1 time(s).
> 2009-03-10 03:14:15,057 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-03-10 03:14:15,057 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> (a,1L,{(hello),(goodbye)})
> {code}
> What I expect for the output of Cfiltered is actually:
> (a,1L,{(hello),(goodbye)})
> (b,0L,{(goodbye)})
> (c,3L,{(hello),(hello),(hello)})
> (d,0L,{(what)})
> The data file is:
> {code}
> a hello
> a goodbye
> b goodbye
> c hello
> c hello
> c hello
> d what
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.