[jira] Commented: (PIG-259) allow store to overwrite existing directroy

["Alan Gates (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829697#action_12829697 ] 

Alan Gates commented on PIG-259:
--------------------------------

A few comments and questions on this:

1) We should make this work against the load/store branch instead of trunk.  We're hoping to merge load/store into trunk in a week or two, so it makes more sense to put it there.  This will also have implications for load/store.  One, it will need to communicate to the new validate function that it's ok if the file (or whatever is being overwritten) exists.  Two, load implementations will need to handle removing the file (or whatever) if necessary.  For example, PigStorage will need to handle removing the file so MR doesn't complain.

2) Should we have overwrite be a keyword (as originally proposed and in the patch) or should it be string, like hints in join?  I don't have a strong opinion one way or another but I think it's worth considering which we want.

3) Is the semantic of overwrite that it saves whether the file is there or not, or that it's an error if the file is not there to write?  Write whether there or not makes more sense to me, but I wanted to make sure we all agree on it.

4) What happens when a user requests overwrite and the job fails before it runs?  In the current implementation the file will be removed up front, so any planning errors will still result in the file being removed.  Also, the file will be removed up front, even if the job remains in Hadoop's queue for a long time waiting to run.  At the very least, I think Pig should delay removing the file until it is ready to launch the job so that type checking errors or whatever don't result in the file being removed when the job is not run.


> allow store to overwrite existing directroy
> -------------------------------------------
>
>                 Key: PIG-259
>                 URL: https://issues.apache.org/jira/browse/PIG-259
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.0
>            Reporter: Olga Natkovich
>            Assignee: Jeff Zhang
>             Fix For: 0.8.0
>
>         Attachments: Pig_259.patch
>
>
> we have users who are asking for a flag to overwrite existing directory

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.