You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Amir Youssefi (JIRA)" <ji...@apache.org> on 2008/07/18 04:09:31 UTC
[jira] Created: (PIG-324) Combiner Error
Combiner Error
--------------
Key: PIG-324
URL: https://issues.apache.org/jira/browse/PIG-324
Project: Pig
Issue Type: Bug
Environment: Pig + Hadoop 17
Reporter: Amir Youssefi
A = load '...' USING PigStorage('\t') AS (c1, c2, c3, n1);
B = group A by (c1,c2,c3);
C = foreach B generate flatten(group), SUM(A.n1);
store C into ...;
Runs with combiner and errors out.
java.io.IOException: For input string: "..." additional info: iteration = 1bag size = 2 partial sum = 0.0
previous tupple = (...)
at org.apache.pig.builtin.SUM.sum(SUM.java:95)
at org.apache.pig.builtin.SUM$Final.exec(SUM.java:63)
at org.apache.pig.builtin.SUM$Final.exec(SUM.java:60)
at org.apache.pig.impl.eval.FuncEvalSpec$1.add(FuncEvalSpec.java:116)
at org.apache.pig.impl.eval.GenerateSpec$CrossProductItem.<init>(GenerateSpec.java:159)
at org.apache.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:79)
at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:165)
at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:80)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:391)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Work-around was that I put out combiner:
C = foreach B generate SUM(A.n1),flatten(group);
and it worked. Input data has some private information in it so I cannot post it. Let me know if it was not possible to solve it without having it. Then we compile a similar input.
c1,c2,c3 are alphabetic,
n1 is numeric.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-324) Combiner Error
Posted by "Amir Youssefi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614870#action_12614870 ]
Amir Youssefi commented on PIG-324:
-----------------------------------
Workaround which disables combiner and makes script run successfully:
A = load '...' USING PigStorage('\t') AS (c1, c2, c3, n1);
B = group A by (c1,c2,c3);
C = foreach B generate SUM(A.n1) , flatten(group);
> Combiner Error
> --------------
>
> Key: PIG-324
> URL: https://issues.apache.org/jira/browse/PIG-324
> Project: Pig
> Issue Type: Bug
> Environment: Pig + Hadoop 17
> Reporter: Amir Youssefi
>
> A = load '...' USING PigStorage('\t') AS (c1, c2, c3, n1);
> B = group A by (c1,c2,c3);
> C = foreach B generate flatten(group), SUM(A.n1);
> store C into ...;
> Runs with combiner and errors out.
> java.io.IOException: For input string: "..." additional info: iteration = 1bag size = 2 partial sum = 0.0
> previous tupple = (...)
> at org.apache.pig.builtin.SUM.sum(SUM.java:95)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:63)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:60)
> at org.apache.pig.impl.eval.FuncEvalSpec$1.add(FuncEvalSpec.java:116)
> at org.apache.pig.impl.eval.GenerateSpec$CrossProductItem.<init>(GenerateSpec.java:159)
> at org.apache.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:79)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:165)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:80)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:391)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
>
> Work-around was that I put out combiner:
> C = foreach B generate SUM(A.n1),flatten(group);
> and it worked. Input data has some private information in it so I cannot post it. Let me know if it was not possible to solve it without having it. Then we compile a similar input.
> c1,c2,c3 are alphabetic,
> n1 is numeric.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-324) Combiner Error
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615421#action_12615421 ]
Olga Natkovich commented on PIG-324:
------------------------------------
we need a reproducible case to look into this. can you provide a fragment of data that is causing this problem?
> Combiner Error
> --------------
>
> Key: PIG-324
> URL: https://issues.apache.org/jira/browse/PIG-324
> Project: Pig
> Issue Type: Bug
> Environment: Pig + Hadoop 17
> Reporter: Amir Youssefi
>
> A = load '...' USING PigStorage('\t') AS (c1, c2, c3, n1);
> B = group A by (c1,c2,c3);
> C = foreach B generate flatten(group), SUM(A.n1);
> store C into ...;
> Runs with combiner and errors out.
> java.io.IOException: For input string: "..." additional info: iteration = 1bag size = 2 partial sum = 0.0
> previous tupple = (...)
> at org.apache.pig.builtin.SUM.sum(SUM.java:95)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:63)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:60)
> at org.apache.pig.impl.eval.FuncEvalSpec$1.add(FuncEvalSpec.java:116)
> at org.apache.pig.impl.eval.GenerateSpec$CrossProductItem.<init>(GenerateSpec.java:159)
> at org.apache.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:79)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:165)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:80)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:391)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
>
> Work-around was that I put out combiner:
> C = foreach B generate SUM(A.n1),flatten(group);
> and it worked. Input data has some private information in it so I cannot post it. Let me know if it was not possible to solve it without having it. Then we compile a similar input.
> c1,c2,c3 are alphabetic,
> n1 is numeric.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-324) Combiner Error
Posted by "Amir Youssefi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615418#action_12615418 ]
Amir Youssefi commented on PIG-324:
-----------------------------------
Yes.
> Combiner Error
> --------------
>
> Key: PIG-324
> URL: https://issues.apache.org/jira/browse/PIG-324
> Project: Pig
> Issue Type: Bug
> Environment: Pig + Hadoop 17
> Reporter: Amir Youssefi
>
> A = load '...' USING PigStorage('\t') AS (c1, c2, c3, n1);
> B = group A by (c1,c2,c3);
> C = foreach B generate flatten(group), SUM(A.n1);
> store C into ...;
> Runs with combiner and errors out.
> java.io.IOException: For input string: "..." additional info: iteration = 1bag size = 2 partial sum = 0.0
> previous tupple = (...)
> at org.apache.pig.builtin.SUM.sum(SUM.java:95)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:63)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:60)
> at org.apache.pig.impl.eval.FuncEvalSpec$1.add(FuncEvalSpec.java:116)
> at org.apache.pig.impl.eval.GenerateSpec$CrossProductItem.<init>(GenerateSpec.java:159)
> at org.apache.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:79)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:165)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:80)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:391)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
>
> Work-around was that I put out combiner:
> C = foreach B generate SUM(A.n1),flatten(group);
> and it worked. Input data has some private information in it so I cannot post it. Let me know if it was not possible to solve it without having it. Then we compile a similar input.
> c1,c2,c3 are alphabetic,
> n1 is numeric.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-324) Combiner Error
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615415#action_12615415 ]
Olga Natkovich commented on PIG-324:
------------------------------------
Is this data dependent? I know I ran similar queries in the past without any problem.
> Combiner Error
> --------------
>
> Key: PIG-324
> URL: https://issues.apache.org/jira/browse/PIG-324
> Project: Pig
> Issue Type: Bug
> Environment: Pig + Hadoop 17
> Reporter: Amir Youssefi
>
> A = load '...' USING PigStorage('\t') AS (c1, c2, c3, n1);
> B = group A by (c1,c2,c3);
> C = foreach B generate flatten(group), SUM(A.n1);
> store C into ...;
> Runs with combiner and errors out.
> java.io.IOException: For input string: "..." additional info: iteration = 1bag size = 2 partial sum = 0.0
> previous tupple = (...)
> at org.apache.pig.builtin.SUM.sum(SUM.java:95)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:63)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:60)
> at org.apache.pig.impl.eval.FuncEvalSpec$1.add(FuncEvalSpec.java:116)
> at org.apache.pig.impl.eval.GenerateSpec$CrossProductItem.<init>(GenerateSpec.java:159)
> at org.apache.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:79)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:165)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:80)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:391)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
>
> Work-around was that I put out combiner:
> C = foreach B generate SUM(A.n1),flatten(group);
> and it worked. Input data has some private information in it so I cannot post it. Let me know if it was not possible to solve it without having it. Then we compile a similar input.
> c1,c2,c3 are alphabetic,
> n1 is numeric.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-324) Combiner Error
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates reassigned PIG-324:
------------------------------
Assignee: Alan Gates
> Combiner Error
> --------------
>
> Key: PIG-324
> URL: https://issues.apache.org/jira/browse/PIG-324
> Project: Pig
> Issue Type: Bug
> Environment: Pig + Hadoop 17
> Reporter: Amir Youssefi
> Assignee: Alan Gates
>
> A = load '...' USING PigStorage('\t') AS (c1, c2, c3, n1);
> B = group A by (c1,c2,c3);
> C = foreach B generate flatten(group), SUM(A.n1);
> store C into ...;
> Runs with combiner and errors out.
> java.io.IOException: For input string: "..." additional info: iteration = 1bag size = 2 partial sum = 0.0
> previous tupple = (...)
> at org.apache.pig.builtin.SUM.sum(SUM.java:95)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:63)
> at org.apache.pig.builtin.SUM$Final.exec(SUM.java:60)
> at org.apache.pig.impl.eval.FuncEvalSpec$1.add(FuncEvalSpec.java:116)
> at org.apache.pig.impl.eval.GenerateSpec$CrossProductItem.<init>(GenerateSpec.java:159)
> at org.apache.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:79)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:165)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:80)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:391)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
>
> Work-around was that I put out combiner:
> C = foreach B generate SUM(A.n1),flatten(group);
> and it worked. Input data has some private information in it so I cannot post it. Let me know if it was not possible to solve it without having it. Then we compile a similar input.
> c1,c2,c3 are alphabetic,
> n1 is numeric.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.