You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Pradeep Kamath (JIRA)" <ji...@apache.org> on 2010/02/26 23:44:06 UTC
[jira] Created: (PIG-1265) Change LoadMetadata and StoreMetadata to
use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
---------------------------------------------------------------------------------------------------------------------------------
Key: PIG-1265
URL: https://issues.apache.org/jira/browse/PIG-1265
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Pradeep Kamath
Assignee: Pradeep Kamath
Fix For: 0.7.0
Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1265) Change LoadMetadata and StoreMetadata
to use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839104#action_12839104 ]
Richard Ding commented on PIG-1265:
-----------------------------------
+1
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1265) Change LoadMetadata and StoreMetadata to
use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1265:
--------------------------------
Status: Patch Available (was: Open)
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265-2.patch, PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1265) Change LoadMetadata and StoreMetadata
to use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839946#action_12839946 ]
Hadoop QA commented on PIG-1265:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12437515/PIG-1265-2.patch
against trunk revision 916793.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 17 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 520 release audit warnings (more than the trunk's current 519 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/217/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/217/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/217/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/217/console
This message is automatically generated.
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265-2.patch, PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1265) Change LoadMetadata and StoreMetadata to
use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1265:
--------------------------------
Attachment: PIG-1265-2.patch
There were some failures in zebra nightly tests which are addressed in the new patch.
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265-2.patch, PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1265) Change LoadMetadata and StoreMetadata to
use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1265:
--------------------------------
Status: Open (was: Patch Available)
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265-2.patch, PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1265) Change LoadMetadata and StoreMetadata to
use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1265:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
The release audit warning is due to changes in html files and not due to any issues in files in the patch.
Patch committed.
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265-2.patch, PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1265) Change LoadMetadata and StoreMetadata to
use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1265:
--------------------------------
Attachment: PIG-1265.patch
Attached patch with changes per description. I have also removed code which used to store the storefunc funcscpec in string form into the conf - this should no longer be needed by StoreFuncs. It was added for StoreFuncs in the earlier release which had an OutputFormat underneath (example zebra) - now that StoreFunc and OutputFormat are closely integrated this should no longer be needed.
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (PIG-1265) Change LoadMetadata and StoreMetadata to
use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai closed PIG-1265.
---------------------------
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265-2.patch, PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1265) Change LoadMetadata and StoreMetadata
to use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839530#action_12839530 ]
Pradeep Kamath commented on PIG-1265:
-------------------------------------
All unit tests succeeded on a local run on my machine.
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1265) Change LoadMetadata and StoreMetadata
to use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839166#action_12839166 ]
Hadoop QA commented on PIG-1265:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12437280/PIG-1265.patch
against trunk revision 916793.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/228/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/228/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/228/console
This message is automatically generated.
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1265) Change LoadMetadata and StoreMetadata
to use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839938#action_12839938 ]
Ashutosh Chauhan commented on PIG-1265:
---------------------------------------
Looked at the diff of first and second patch. +1 for that part.
One thing unrelated to patch I want to highlight is: these kind of issues creeps in because our api is too wide, there are multiple ways of getting same thing done in Pig (executing script through Java api in this case), each exercising different code paths. So, first thing we need is to establish what is part of our api and what is meant for Pig's internal purposes only? Is every single public method part of api ? Or, only those which have properly documented Javadocs are supported ? Or only those documented in wiki page ?
First we establish what constitutes our public api, then we should systematically start decreasing the visibility of those methods which are public but not meant as an api, then deprecate them and eventually remove them.
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265-2.patch, PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1265) Change LoadMetadata and StoreMetadata to
use Job instead of Configuraiton and add a cleanupOnFailure method to
StoreFuncInterface
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1265:
--------------------------------
Assignee: Pradeep Kamath (was: Pradeep Kamath)
Status: Patch Available (was: Open)
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration - for example InputFormat/OutputFormat implementations use Job to store input/output location. So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores in the job) are hdfs locations and attempts to delete them. Since output locations could be non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation - hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a default implementation should be provided in the StoreFunc abstract class which checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.