You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Pradeep Kamath (JIRA)" <ji...@apache.org> on 2009/10/29 22:55:59 UTC

[jira] Created: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
----------------------------------------------------------------------------------------------

                 Key: PIG-1063
                 URL: https://issues.apache.org/jira/browse/PIG-1063
             Project: Pig
          Issue Type: Bug
    Affects Versions: 0.4.0
            Reporter: Pradeep Kamath
            Assignee: Pradeep Kamath


A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
{noformat}
a = load 'input.txt';
b = filter a by $0 < 10;
store b into 'output1' using StoreWithOutputFormat();
c = group a by $0;
d = foreach c generate group, COUNT(a.$0);
store d into 'output2' using StoreWithOutputFormat();
{noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772072#action_12772072 ] 

Olga Natkovich commented on PIG-1063:
-------------------------------------

+1; changes look good

> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1063
>                 URL: https://issues.apache.org/jira/browse/PIG-1063
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>         Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pradeep Kamath updated PIG-1063:
--------------------------------

    Status: Patch Available  (was: Open)

> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1063
>                 URL: https://issues.apache.org/jira/browse/PIG-1063
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>         Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pradeep Kamath updated PIG-1063:
--------------------------------

    Attachment: PIG-1063.patch

Attached patch fixes the issue. The fix is in PigOutputFormat.checkOutputSpecs(), we now check if there are any stores in the map or reduce plan which provide a non null OutputFormat as return value of  getStorePreparationClass(). In this case, the checkOutputSpecs() method on the OutputFormat of the store is called.

> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1063
>                 URL: https://issues.apache.org/jira/browse/PIG-1063
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>         Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pradeep Kamath updated PIG-1063:
--------------------------------

    Status: Patch Available  (was: Open)

The patch will give one extra javac deprecation warning due to a hadoop deprecated class which cannot be suppressed in code due to a javac [bug|http://bugs.sun.com/view_bug.do?bug_id=6594914] - this would need to be resolved when we move to new hadoop api.

> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1063
>                 URL: https://issues.apache.org/jira/browse/PIG-1063
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>         Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pradeep Kamath updated PIG-1063:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.6.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Patch committed to trunk

> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1063
>                 URL: https://issues.apache.org/jira/browse/PIG-1063
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>             Fix For: 0.6.0
>
>         Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772068#action_12772068 ] 

Hadoop QA commented on PIG-1063:
--------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12423638/PIG-1063.patch
  against trunk revision 831169.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 199 javac compiler warnings (more than the trunk's current 198 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/133/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/133/console

This message is automatically generated.

> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1063
>                 URL: https://issues.apache.org/jira/browse/PIG-1063
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>         Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pradeep Kamath updated PIG-1063:
--------------------------------

    Status: Open  (was: Patch Available)

> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1063
>                 URL: https://issues.apache.org/jira/browse/PIG-1063
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>         Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771846#action_12771846 ] 

Hadoop QA commented on PIG-1063:
--------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12423638/PIG-1063.patch
  against trunk revision 831169.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 199 javac compiler warnings (more than the trunk's current 198 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/131/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/131/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/131/console

This message is automatically generated.

> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1063
>                 URL: https://issues.apache.org/jira/browse/PIG-1063
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>         Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1063) Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case

Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771985#action_12771985 ] 

Pradeep Kamath commented on PIG-1063:
-------------------------------------

The 1 javac warning is due to the deprecation warning explained in my previous comment. The unit test failure seems unrelated and looks like a temporary env. issue - resubmitting to check if the tests pass now.

> Pig does not call checkOutSpecs() on OutputFormat provided by StoreFunc in the multistore case
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1063
>                 URL: https://issues.apache.org/jira/browse/PIG-1063
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>         Attachments: PIG-1063.patch
>
>
> A StoreFunc implementation can inform pig of an OutputFormat it uses through the getStoragePreparationClass() method. In a query with multiple stores which gets optimized into a single mapred job, Pig does not call the checkOutputSpecs() method on the outputformat. An example of such a script is:
> {noformat}
> a = load 'input.txt';
> b = filter a by $0 < 10;
> store b into 'output1' using StoreWithOutputFormat();
> c = group a by $0;
> d = foreach c generate group, COUNT(a.$0);
> store d into 'output2' using StoreWithOutputFormat();
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.