You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Pradeep Kamath (JIRA)" <ji...@apache.org> on 2009/10/29 22:55:59 UTC
[jira] Created: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
----------------------------------------------------------------------------------------------
Key: PIG-1063
URL: https://issues.apache.org/jira/browse/PIG-1063
Project: Pig
Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Pradeep Kamath
Assignee: Pradeep Kamath
A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
{noformat}
a = load 'input.txt';
b = filter a by $0 < 10;
store b into 'output1' using StoreWithOutputFormat();
c = group a by $0;
d = foreach c generate group, COUNT(a.$0);
store d into 'output2' using StoreWithOutputFormat();
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772072#action_12772072 ]
Olga Natkovich commented on PIG-1063:
-------------------------------------
+1; changes look good
> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1063
> URL: https://issues.apache.org/jira/browse/PIG-1063
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1063:
--------------------------------
Status: Patch Available (was: Open)
> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1063
> URL: https://issues.apache.org/jira/browse/PIG-1063
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1063:
--------------------------------
Attachment: PIG-1063.patch
Attached patch fixes the issue. The fix is in PigOutputFormat.checkOutputSpecs(), we now check if there are any stores in the map or reduce plan which provide a non null OutputFormat as return value of getStorePreparationClass(). In this case, the checkOutputSpecs() method on the OutputFormat of the store is called.
> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1063
> URL: https://issues.apache.org/jira/browse/PIG-1063
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1063:
--------------------------------
Status: Patch Available (was: Open)
The patch will give one extra javac deprecation warning due to a hadoop deprecated class which cannot be suppressed in code due to a javac [bug|http://bugs.sun.com/view_bug.do?bug_id=6594914] - this would need to be resolved when we move to new hadoop api.
> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1063
> URL: https://issues.apache.org/jira/browse/PIG-1063
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1063:
--------------------------------
Resolution: Fixed
Fix Version/s: 0.6.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Patch committed to trunk
> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1063
> URL: https://issues.apache.org/jira/browse/PIG-1063
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.6.0
>
> Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772068#action_12772068 ]
Hadoop QA commented on PIG-1063:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423638/PIG-1063.patch
against trunk revision 831169.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 199 javac compiler warnings (more than the trunk's current 198 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/133/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/133/console
This message is automatically generated.
> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1063
> URL: https://issues.apache.org/jira/browse/PIG-1063
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1063:
--------------------------------
Status: Open (was: Patch Available)
> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1063
> URL: https://issues.apache.org/jira/browse/PIG-1063
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771846#action_12771846 ]
Hadoop QA commented on PIG-1063:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423638/PIG-1063.patch
against trunk revision 831169.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 199 javac compiler warnings (more than the trunk's current 198 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/131/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/131/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/131/console
This message is automatically generated.
> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1063
> URL: https://issues.apache.org/jira/browse/PIG-1063
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1063) Pig does not call checkOutSpecs() on
OutputFormat provided by StoreFunc in the multistore case
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771985#action_12771985 ]
Pradeep Kamath commented on PIG-1063:
-------------------------------------
The 1 javac warning is due to the deprecation warning explained in my previous comment. The unit test failure seems unrelated and looks like a temporary env. issue - resubmitting to check if the tests pass now.
> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1063
> URL: https://issues.apache.org/jira/browse/PIG-1063
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.