You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ankur (JIRA)" <ji...@apache.org> on 2010/01/15 08:47:54 UTC
[jira] Created: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
POCast throws exception for certain sequences of LOAD, FILTER, FORACH
---------------------------------------------------------------------
Key: PIG-1191
URL: https://issues.apache.org/jira/browse/PIG-1191
Project: Pig
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Ankur
Priority: Blocker
When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
...
Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801292#action_12801292 ]
Hadoop QA commented on PIG-1191:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430495/PIG-1191-2.patch
against trunk revision 899502.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 479 release audit warnings (more than the trunk's current 478 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/181/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/181/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/181/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/181/console
This message is automatically generated.
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Attachments: PIG-1191-1.patch, PIG-1191-2.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1191:
----------------------------
Attachment: PIG-1191-1.patch
Hi, Ankur,
Can you check if this patch works?
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Priority: Blocker
> Attachments: PIG-1191-1.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802413#action_12802413 ]
Alan Gates commented on PIG-1191:
---------------------------------
+1, looks good.
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Attachments: PIG-1191-1.patch, PIG-1191-2.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801687#action_12801687 ]
Ankur commented on PIG-1191:
----------------------------
Verified that PIG-1191-2.patch successfully passes all 6 test cases with expected results.
So barring the increased number of release audit warnings,
+1 for commit
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Attachments: PIG-1191-1.patch, PIG-1191-2.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1191:
--------------------------------
Status: Patch Available (was: Open)
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Attachments: PIG-1191-1.patch, PIG-1191-2.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1191:
----------------------------
Resolution: Fixed
Fix Version/s: 0.6.0
Status: Resolved (was: Patch Available)
Checked into trunk for Pradeep, since he's out.
As this is significant bug, I intend to apply this to the 0.6 branch as well.
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Fix For: 0.6.0
>
> Attachments: PIG-1191-1.patch, PIG-1191-2.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800636#action_12800636 ]
Ankur commented on PIG-1191:
----------------------------
Case 1, 2: Succeeds
Case 3 : Fails
Case 4,5: Empty results. Both of them are using consecutive projection of complex fields.
I'll add 1 more test case
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Priority: Blocker
> Attachments: PIG-1191-1.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800655#action_12800655 ]
Ankur commented on PIG-1191:
----------------------------
CASE 6: In CASE 1 replace LIMIT with a GROUP BY followed by FOREACH
============================================================
Succeeds with the given patch.
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Priority: Blocker
> Attachments: PIG-1191-1.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1191:
--------------------------------
Attachment: PIG-1191-2.patch
Attached patch to fix the issues in the script reported. The cause for the issue is that in the parser, getFieldSchema is called on Cast's expression which results in lineage information (canonical map) being updated in the inner map look up operator in the case of two level map look ups. However later the inner plans of foreach get cloned invalidating the previously stored lineage information. In the TypeCheckingVisitor , while visiting LOForeach, the schemas of all operators in the inner plans is supposed to be reset using SchemaRemover so that schemas get re-calculated. However SchemaRemover did not have many of the visit() methods implemented - this patch implements the required methods and fixes the issues reported here.
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Attachments: PIG-1191-1.patch, PIG-1191-2.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800609#action_12800609 ]
Ankur commented on PIG-1191:
----------------------------
Listed below are the identified cases.
CASE 1: LOAD -> FILTER -> FOREACH -> LIMIT -> STORE
===================================================
SCRIPT
-----------
sds = LOAD '/my/data/location'
USING my.org.MyMapLoader()
AS (simpleFields:map[], mapFields:map[], listMapFields:map[]);
queries = FILTER sds BY mapFields#'page_params'#'query' is NOT NULL;
queries_rand = FOREACH queries
GENERATE (CHARARRAY) (mapFields#'page_params'#'query') AS query_string;
queries_limit = LIMIT queries_rand 100;
STORE queries_limit INTO 'out';
RESULT
------------
FAILS in reduce stage with the following exception
org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine
how to convert the bytearray to string.
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:364)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:288)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:423)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:391)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:371)
CASE 2: LOAD -> FOREACH -> FILTER -> LIMIT -> STORE
===================================================
Note that FILTER and FOREACH order is reversed
SCRIPT
-----------
sds = LOAD '/my/data/location'
USING my.org.MyMapLoader()
AS (simpleFields:map[], mapFields:map[], listMapFields:map[]);
queries_rand = FOREACH sds
GENERATE (CHARARRAY) (mapFields#'page_params'#'query') AS query_string;
queries = FILTER queries_rand BY query_string IS NOT null;
queries_limit = LIMIT queries 100;
STORE queries_limit INTO 'out';
RESULT
-----------
SUCCESS - Results are correctly stored. So if a projection is done before FILTER it recieves the LoadFunc in the POCast
operator and everything is cool.
CASE 3: LOAD -> FOREACH -> FOREACH -> FILTER -> LIMIT -> STORE
==============================================================
SCRIPT
-----------
ds = LOAD '/my/data/location'
USING my.org.MyMapLoader()
AS (simpleFields:map[], mapFields:map[], listMapFields:map[]);
params = FOREACH sds GENERATE
(map[]) (mapFields#'page_params') AS params;
queries = FOREACH params
GENERATE (CHARARRAY) (params#'query') AS query_string;
queries_filtered = FILTER queries
BY query_string IS NOT null;
queries_limit = LIMIT queries_filtered 100;
STORE queries_limit INTO 'out';
RESULT
-----------
FAILS in Map stage. Looks like the 2nd FOREACH did not get the loadFunc and bailed out with following stack trace
org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine
how to convert the bytearray to string. at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639) at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:364)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:288)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.getNext(POFilter.java:95)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLimit.getNext(POLimit.java:85) at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:256)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at
CASE 4: LOAD -> FOREACH -> FOREACH -> LIMIT -> STORE
====================================================
SCRIPT
-----------
sds = LOAD '/my/data/location'
USING my.org.MyMapLoader()
AS (simpleFields:map[], mapFields:map[], listMapFields:map[]);
params = FOREACH sds GENERATE
(map[]) (mapFields#'page_params') AS params;
queries = FOREACH params
GENERATE (CHARARRAY) (params#'query') AS query_string;
queries_limit = LIMIT queries 100;
STORE queries_limit INTO 'out';
RESULT
-----------
SUCCESS. The two FOREACH seem to be getting the loadFunc.
CASE 5: LOAD -> FOREACH -> FOREACH -> FOREACH -> LIMIT -> STORE
================================================================
SCRIPT
-----------
ds = LOAD '/my/data/location'
USING my.org.MyMapLoader()
AS (simpleFields:map[], mapFields:map[], listMapFields:map[]);
params = FOREACH sds GENERATE
(map[]) (mapFields#'page_params') AS params;
queries = FOREACH params
GENERATE (CHARARRAY) (params#'query') AS query_string;
rand_queries = FOREACH queries GENERATE query_string as query;
queries_limit = LIMIT rand_queries 100;
STORE rand_queries INTO 'out';
RESULT
-----------
FAILS in map stage. Again the poor second FOREACH seems to be bailing out with stack trace
org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine
how to convert the bytearray to string. at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639) at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:364)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:288)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253)
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Priority: Blocker
> Attachments: PIG-1191-1.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1191:
--------------------------------
Status: Open (was: Patch Available)
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Attachments: PIG-1191-1.patch, PIG-1191-2.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1191:
----------------------------
Status: Patch Available (was: Open)
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Priority: Blocker
> Attachments: PIG-1191-1.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802931#action_12802931 ]
Alan Gates commented on PIG-1191:
---------------------------------
Checked into 0.6 branch.
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Fix For: 0.6.0
>
> Attachments: PIG-1191-1.patch, PIG-1191-2.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankur reassigned PIG-1191:
--------------------------
Assignee: Pradeep Kamath
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Attachments: PIG-1191-1.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800610#action_12800610 ]
Ankur commented on PIG-1191:
----------------------------
I'll check and update the ticket
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Priority: Blocker
> Attachments: PIG-1191-1.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800678#action_12800678 ]
Hadoop QA commented on PIG-1191:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430376/PIG-1191-1.patch
against trunk revision 899502.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/176/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/176/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/176/console
This message is automatically generated.
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Attachments: PIG-1191-1.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1191) POCast throws exception for certain
sequences of LOAD, FILTER, FORACH
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800660#action_12800660 ]
Ankur commented on PIG-1191:
----------------------------
Small correct in comment dated - 15/Jan/10 09:39 AM
Case 5: Still FAILS
> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>
> Key: PIG-1191
> URL: https://issues.apache.org/jira/browse/PIG-1191
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ankur
> Assignee: Pradeep Kamath
> Priority: Blocker
> Attachments: PIG-1191-1.patch
>
>
> When using a custom load/store function, one that returns complex data (map of maps, list of maps), for certain sequences of LOAD, FILTER, FOREACH pig script throws an exception of the form -
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right load function for doing the conversion and consequently bailed out with the exception failing the entire pig script.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.