You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2008/06/17 02:31:44 UTC
[jira] Created: (PIG-272) Failure running complex script with
streaming
Failure running complex script with streaming
---------------------------------------------
Key: PIG-272
URL: https://issues.apache.org/jira/browse/PIG-272
Project: Pig
Issue Type: Bug
Reporter: Olga Natkovich
The following script fails (stack is further down):
define CMD `perl identity.pl`;
define CMD1 `perl identity.pl`;
A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
B = stream A through CMD;
store B into 'B1';
C = stream B through CMD1;
D = JOIN B by name, C by name;
store D into 'D1';
If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-272) Failure running complex script with
streaming
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich reassigned PIG-272:
----------------------------------
Assignee: Arun C Murthy
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-272) Failure running complex script with
streaming
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12607292#action_12607292 ]
Olga Natkovich commented on PIG-272:
------------------------------------
Thanks, Arun. I will be testing your patch today.
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-272_0_20080621.patch, PIG-272_test.pig, split.pl
>
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-272) Failure running complex script with
streaming
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606546#action_12606546 ]
Arun C Murthy commented on PIG-272:
-----------------------------------
Sigh, attaching the original store function isn't enough.
The problem is that currently Pig re-runs the entire job and doesn't use the existing results on HDFS for the JOIN in the above example. When that happens the StreamingCommand's output-spec is still setup as 'BinaryStorage' and results in this error.
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-272) Failure running complex script with
streaming
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605781#action_12605781 ]
Olga Natkovich commented on PIG-272:
------------------------------------
Arun, helped to diagnose the problem. The issue is that the following sequence
B = stream A through CMD;
store B into 'B1';
kicks in the optimization and as the result store users BinaryStorage to write the results of the first job.
When the second job starts to run, it realizes that it can reuse the results and tries to load them also using BinaryStorage which is wrong and causes exceptions since the tuples don't have structure expected by the second script.
The solution is to attach the original store function to the materialized results; however, the code changes for it are quite ugly.
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-272) Failure running complex script with
streaming
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-272:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I committed the changes. I ran all existing unit and end-to-end tests as well as the end-to-end tests provided by Arun. They all passed.
Thanks, Arun for fixing this issue.
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-272_0_20080621.patch, PIG-272_test.pig, split.pl
>
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-272) Failure running complex
script with streaming
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606546#action_12606546 ]
acmurthy edited comment on PIG-272 at 6/19/08 1:49 PM:
------------------------------------------------------------
Sigh, attaching the original store function isn't enough.
The problem is that currently Pig re-executes the entire pipeline and doesn't use the existing results on HDFS for the JOIN in the above example. When that happens the StreamingCommand's output-spec is still setup as 'BinaryStorage' and results in this error.
was (Author: acmurthy):
Sigh, attaching the original store function isn't enough.
The problem is that currently Pig re-executes the entire and doesn't use the existing results on HDFS for the JOIN in the above example. When that happens the StreamingCommand's output-spec is still setup as 'BinaryStorage' and results in this error.
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-272) Failure running complex script with
streaming
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-272:
------------------------------
Status: Patch Available (was: Open)
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-272_0_20080621.patch, PIG-272_test.pig, split.pl
>
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-272) Failure running complex script with
streaming
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606663#action_12606663 ]
Arun C Murthy commented on PIG-272:
-----------------------------------
To clarify the above comment: it seems like the 'materialized results' of the first of the two resulting Map-Reduce jobs isn't being used by the second. Rather, it goes ahead and re-executes the entire pipeline. Clearly, it is rather inefficient. Thus, it looks like the existing code for tracking/using previous job's results has a bug.
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-272) Failure running complex
script with streaming
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606546#action_12606546 ]
acmurthy edited comment on PIG-272 at 6/19/08 1:48 PM:
------------------------------------------------------------
Sigh, attaching the original store function isn't enough.
The problem is that currently Pig re-executes the entire and doesn't use the existing results on HDFS for the JOIN in the above example. When that happens the StreamingCommand's output-spec is still setup as 'BinaryStorage' and results in this error.
was (Author: acmurthy):
Sigh, attaching the original store function isn't enough.
The problem is that currently Pig re-runs the entire job and doesn't use the existing results on HDFS for the JOIN in the above example. When that happens the StreamingCommand's output-spec is still setup as 'BinaryStorage' and results in this error.
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-272) Failure running complex script with
streaming
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-272:
------------------------------
Attachment: split.pl
PIG-272_test.pig
PIG-272_0_20080621.patch
Attached fix. The patch ensures we deep-copy the StreamingCommand before optimizing it and reverts the optimization piece-meal (i.e for input and output separately).
The test cases are quite complex/convoluted and are pretty hard to convert to unit-tests, which I why I've attached them here and propose we integrate them into our end-to-end tests...
> Failure running complex script with streaming
> ---------------------------------------------
>
> Key: PIG-272
> URL: https://issues.apache.org/jira/browse/PIG-272
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-272_0_20080621.patch, PIG-272_test.pig, split.pl
>
>
> The following script fails (stack is further down):
> define CMD `perl identity.pl`;
> define CMD1 `perl identity.pl`;
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through CMD;
> store B into 'B1';
> C = stream B through CMD1;
> D = JOIN B by name, C by name;
> store D into 'D1';
> If I remove the intermediate store, the script works fine. Also if I replace streaming commands with other operators such as filter and foreach, it works even with the intermediate store.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.