You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Richard Ding (JIRA)" <ji...@apache.org> on 2010/07/30 00:00:21 UTC
[jira] Created: (PIG-1525) Incorrect data generated by diff of SUM
Incorrect data generated by diff of SUM
---------------------------------------
Key: PIG-1525
URL: https://issues.apache.org/jira/browse/PIG-1525
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Richard Ding
Assignee: Richard Ding
Fix For: 0.8.0
Given data;
input1:
{code}
id9 0
{code}
input2:
{code}
id8 1
id9 1
{code}
Pig script
{code}
A = LOAD 'input1' AS (id:chararray, val:long);
B = LOAD 'input2' AS (id:chararray, val:long);
C = COGROUP A BY id, B BY id;
D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
dump D;
{code}
generates incorrect data:
{code}
(id8,1L,,)
(id9,1L,0L,-2L)
{code}
The workaround is to replace the FOREACH statement with
{code}
D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
E = FOREACH D GENERATE $0, b, a, (a-b);
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1525) Incorrect data generated by diff of SUM
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1525:
------------------------------
Status: Patch Available (was: Open)
> Incorrect data generated by diff of SUM
> ---------------------------------------
>
> Key: PIG-1525
> URL: https://issues.apache.org/jira/browse/PIG-1525
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1525.patch
>
>
> Given data;
> input1:
> {code}
> id9 0
> {code}
> input2:
> {code}
> id8 1
> id9 1
> {code}
> Pig script
> {code}
> A = LOAD 'input1' AS (id:chararray, val:long);
> B = LOAD 'input2' AS (id:chararray, val:long);
> C = COGROUP A BY id, B BY id;
> D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
> dump D;
> {code}
> generates incorrect data:
> {code}
> (id8,1L,,)
> (id9,1L,0L,-2L)
> {code}
> The workaround is to replace the FOREACH statement with
> {code}
> D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
> E = FOREACH D GENERATE $0, b, a, (a-b);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1525) Incorrect data generated by diff of SUM
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1525:
------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
> Incorrect data generated by diff of SUM
> ---------------------------------------
>
> Key: PIG-1525
> URL: https://issues.apache.org/jira/browse/PIG-1525
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1525.patch, PIG-1525_1.patch
>
>
> Given data;
> input1:
> {code}
> id9 0
> {code}
> input2:
> {code}
> id8 1
> id9 1
> {code}
> Pig script
> {code}
> A = LOAD 'input1' AS (id:chararray, val:long);
> B = LOAD 'input2' AS (id:chararray, val:long);
> C = COGROUP A BY id, B BY id;
> D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
> dump D;
> {code}
> generates incorrect data:
> {code}
> (id8,1L,,)
> (id9,1L,0L,-2L)
> {code}
> The workaround is to replace the FOREACH statement with
> {code}
> D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
> E = FOREACH D GENERATE $0, b, a, (a-b);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1525) Incorrect data generated by diff of
SUM
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896121#action_12896121 ]
Richard Ding commented on PIG-1525:
-----------------------------------
It turns out that the problem also affects the conditional operator (BinCond).
> Incorrect data generated by diff of SUM
> ---------------------------------------
>
> Key: PIG-1525
> URL: https://issues.apache.org/jira/browse/PIG-1525
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1525.patch
>
>
> Given data;
> input1:
> {code}
> id9 0
> {code}
> input2:
> {code}
> id8 1
> id9 1
> {code}
> Pig script
> {code}
> A = LOAD 'input1' AS (id:chararray, val:long);
> B = LOAD 'input2' AS (id:chararray, val:long);
> C = COGROUP A BY id, B BY id;
> D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
> dump D;
> {code}
> generates incorrect data:
> {code}
> (id8,1L,,)
> (id9,1L,0L,-2L)
> {code}
> The workaround is to replace the FOREACH statement with
> {code}
> D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
> E = FOREACH D GENERATE $0, b, a, (a-b);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1525) Incorrect data generated by diff of SUM
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1525:
------------------------------
Attachment: PIG-1525.patch
> Incorrect data generated by diff of SUM
> ---------------------------------------
>
> Key: PIG-1525
> URL: https://issues.apache.org/jira/browse/PIG-1525
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1525.patch
>
>
> Given data;
> input1:
> {code}
> id9 0
> {code}
> input2:
> {code}
> id8 1
> id9 1
> {code}
> Pig script
> {code}
> A = LOAD 'input1' AS (id:chararray, val:long);
> B = LOAD 'input2' AS (id:chararray, val:long);
> C = COGROUP A BY id, B BY id;
> D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
> dump D;
> {code}
> generates incorrect data:
> {code}
> (id8,1L,,)
> (id9,1L,0L,-2L)
> {code}
> The workaround is to replace the FOREACH statement with
> {code}
> D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
> E = FOREACH D GENERATE $0, b, a, (a-b);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1525) Incorrect data generated by diff of
SUM
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896094#action_12896094 ]
Richard Ding commented on PIG-1525:
-----------------------------------
Results of running test-patch:
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
> Incorrect data generated by diff of SUM
> ---------------------------------------
>
> Key: PIG-1525
> URL: https://issues.apache.org/jira/browse/PIG-1525
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1525.patch
>
>
> Given data;
> input1:
> {code}
> id9 0
> {code}
> input2:
> {code}
> id8 1
> id9 1
> {code}
> Pig script
> {code}
> A = LOAD 'input1' AS (id:chararray, val:long);
> B = LOAD 'input2' AS (id:chararray, val:long);
> C = COGROUP A BY id, B BY id;
> D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
> dump D;
> {code}
> generates incorrect data:
> {code}
> (id8,1L,,)
> (id9,1L,0L,-2L)
> {code}
> The workaround is to replace the FOREACH statement with
> {code}
> D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
> E = FOREACH D GENERATE $0, b, a, (a-b);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1525) Incorrect data generated by diff of
SUM
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896123#action_12896123 ]
Richard Ding commented on PIG-1525:
-----------------------------------
The cause is the interaction between Accumulator UDF and binary operators. In the failure cases, the state kept by Accumulator is not reset cross record boundaries.
> Incorrect data generated by diff of SUM
> ---------------------------------------
>
> Key: PIG-1525
> URL: https://issues.apache.org/jira/browse/PIG-1525
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1525.patch
>
>
> Given data;
> input1:
> {code}
> id9 0
> {code}
> input2:
> {code}
> id8 1
> id9 1
> {code}
> Pig script
> {code}
> A = LOAD 'input1' AS (id:chararray, val:long);
> B = LOAD 'input2' AS (id:chararray, val:long);
> C = COGROUP A BY id, B BY id;
> D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
> dump D;
> {code}
> generates incorrect data:
> {code}
> (id8,1L,,)
> (id9,1L,0L,-2L)
> {code}
> The workaround is to replace the FOREACH statement with
> {code}
> D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
> E = FOREACH D GENERATE $0, b, a, (a-b);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1525) Incorrect data generated by diff of
SUM
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894110#action_12894110 ]
Richard Ding commented on PIG-1525:
-----------------------------------
This problem affects other BinaryExpressionOperators as well.
> Incorrect data generated by diff of SUM
> ---------------------------------------
>
> Key: PIG-1525
> URL: https://issues.apache.org/jira/browse/PIG-1525
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.8.0
>
>
> Given data;
> input1:
> {code}
> id9 0
> {code}
> input2:
> {code}
> id8 1
> id9 1
> {code}
> Pig script
> {code}
> A = LOAD 'input1' AS (id:chararray, val:long);
> B = LOAD 'input2' AS (id:chararray, val:long);
> C = COGROUP A BY id, B BY id;
> D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
> dump D;
> {code}
> generates incorrect data:
> {code}
> (id8,1L,,)
> (id9,1L,0L,-2L)
> {code}
> The workaround is to replace the FOREACH statement with
> {code}
> D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
> E = FOREACH D GENERATE $0, b, a, (a-b);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1525) Incorrect data generated by diff of
SUM
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896637#action_12896637 ]
Thejas M Nair commented on PIG-1525:
------------------------------------
+1
> Incorrect data generated by diff of SUM
> ---------------------------------------
>
> Key: PIG-1525
> URL: https://issues.apache.org/jira/browse/PIG-1525
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1525.patch, PIG-1525_1.patch
>
>
> Given data;
> input1:
> {code}
> id9 0
> {code}
> input2:
> {code}
> id8 1
> id9 1
> {code}
> Pig script
> {code}
> A = LOAD 'input1' AS (id:chararray, val:long);
> B = LOAD 'input2' AS (id:chararray, val:long);
> C = COGROUP A BY id, B BY id;
> D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
> dump D;
> {code}
> generates incorrect data:
> {code}
> (id8,1L,,)
> (id9,1L,0L,-2L)
> {code}
> The workaround is to replace the FOREACH statement with
> {code}
> D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
> E = FOREACH D GENERATE $0, b, a, (a-b);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1525) Incorrect data generated by diff of SUM
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1525:
------------------------------
Attachment: PIG-1525_1.patch
Thanks Thejas for suggesting a simple fix. The new patch passed core tests.
> Incorrect data generated by diff of SUM
> ---------------------------------------
>
> Key: PIG-1525
> URL: https://issues.apache.org/jira/browse/PIG-1525
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1525.patch, PIG-1525_1.patch
>
>
> Given data;
> input1:
> {code}
> id9 0
> {code}
> input2:
> {code}
> id8 1
> id9 1
> {code}
> Pig script
> {code}
> A = LOAD 'input1' AS (id:chararray, val:long);
> B = LOAD 'input2' AS (id:chararray, val:long);
> C = COGROUP A BY id, B BY id;
> D = FOREACH C GENERATE group, SUM(B.val), SUM(A.val), (SUM(A.val) - SUM(B.val));
> dump D;
> {code}
> generates incorrect data:
> {code}
> (id8,1L,,)
> (id9,1L,0L,-2L)
> {code}
> The workaround is to replace the FOREACH statement with
> {code}
> D = FOREACH C GENERATE group, SUM(B.val) as b, SUM(A.val) as a;
> E = FOREACH D GENERATE $0, b, a, (a-b);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.