You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2008/01/31 22:49:12 UTC

[jira] Created: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

creating a file in hdfs should not automatically create the parent directories
------------------------------------------------------------------------------

                 Key: HADOOP-2759
                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
             Project: Hadoop Core
          Issue Type: Bug
          Components: dfs
            Reporter: Owen O'Malley
             Fix For: 0.17.0


I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated HADOOP-2759:
----------------------------

    Affects Version/s: 0.17.0
               Status: Patch Available  (was: Open)

The patch

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-2759_1.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567082#action_12567082 ] 

Hadoop QA commented on HADOOP-2759:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12375077/HADOOP-2759_1.patch
against trunk revision 619744.

    @author +1.  The patch does not contain any @author tags.

    tests included -1.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no tests are needed for this patch.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new javac compiler warnings.

    release audit +1.  The applied patch does not generate any new release audit warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests -1.  The patch failed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1762/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1762/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1762/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1762/console

This message is automatically generated.

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-2759_1.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567307#action_12567307 ] 

pi_song edited comment on HADOOP-2759 at 2/9/08 5:33 AM:
---------------------------------------------------------

This issue involves two things

- data file creation
- directory creation

At the moment if you create any of them, all the parents will be created!!!

I plan to add new mkdir and createFile that don't create all the parent dirs instead of changing the old ones because I don't want to have backward compatibility problems.  Now the question is should I mark the old mkdir and createFile as @deprecated? I still feel sometimes we need methods that ensure parent dir creation but then if I don't it will encourage people to still use the old ones and here comes problems like in HADOOP-2391.

I also need a new name for createFile that doesn't create parents. (For mkdir it's easy, the old one is mkdirs and the new one is mkdir therefore like in Java)

      was (Author: pi_song):
    This issue involves two things

- data file creation
- directory creation

I plan to add new mkdir and createFile that don't create all the parent dirs instead of changing the old ones because I don't want to have backward compatibility problems.  Now the question is should I mark the old mkdir and createFile as @deprecated? I still feel sometimes we need methods that ensure parent dir creation but then if I don't it will encourage people to still use the old ones and here comes problems like in HADOOP-2391.

I also need a new name for createFile that doesn't create parents. (For mkdir it's easy, the old one is mkdirs and the new one is mkdir therefore like in Java)
  
> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-2759_1.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567307#action_12567307 ] 

Pi Song commented on HADOOP-2759:
---------------------------------

This issue involves two things

- data file creation
- directory creation

I plan to add new mkdir and createFile that don't create all the parent dirs instead of changing the old ones because I don't want to have backward compatibility problems.  Now the question is should I mark the old mkdir and createFile as @deprecated? I still feel sometimes we need methods that ensure parent dir creation but then if I don't it will encourage people to still use the old ones and here comes problems like in HADOOP-2391.

I also need a new name for createFile that doesn't create parents. (For mkdir it's easy, the old one is mkdirs and the new one is mkdir therefore like in Java)

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-2759_1.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2759:
-------------------------------------

    Status: Open  (was: Patch Available)

I am canceling this patch because it does not merge with trunk and also because this is a *big* api change. Can somebody please elaborate a use case on why behavour change is required?

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>            Assignee: Pi Song
>             Fix For: 0.18.0
>
>         Attachments: hadoop-2759-complete1.patch, HADOOP-2759_1.patch, hadoop_tmp.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sameer Paranjpye updated HADOOP-2759:
-------------------------------------

    Fix Version/s:     (was: 0.17.0)
                   0.18.0

We should disseminate and build consensus before making this big an API change. Pushing to 0.18 

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.18.0
>
>         Attachments: hadoop-2759-complete1.patch, HADOOP-2759_1.patch, hadoop_tmp.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mukund Madhugiri updated HADOOP-2759:
-------------------------------------

    Fix Version/s:     (was: 0.18.0)

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>            Assignee: Pi Song
>         Attachments: hadoop-2759-complete1.patch, HADOOP-2759_1.patch, hadoop_tmp.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley reassigned HADOOP-2759:
-------------------------------------

    Assignee: Pi Song

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>            Assignee: Pi Song
>             Fix For: 0.18.0
>
>         Attachments: hadoop-2759-complete1.patch, HADOOP-2759_1.patch, hadoop_tmp.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated HADOOP-2759:
----------------------------

    Attachment: hadoop-2759-complete1.patch

This patch introduces

- createFileNoParentPath
- mkdir 

in FileSystem class

Test case also included.
All unit tests look good.

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: hadoop-2759-complete1.patch, HADOOP-2759_1.patch, hadoop_tmp.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated HADOOP-2759:
----------------------------

    Comment: was deleted

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-2759_1.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated HADOOP-2759:
----------------------------

    Attachment: HADOOP-2759_1.patch

The patch

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-2759_1.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated HADOOP-2759:
----------------------------

    Attachment: hadoop_tmp.patch

This is not done yet (only mkdir is done). I'm totally new. If someone please review my code, that would be very nice : D

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-2759_1.patch, hadoop_tmp.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated HADOOP-2759:
----------------------------

    Comment: was deleted

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-2759_1.patch, hadoop_tmp.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595989#action_12595989 ] 

Amareshwari Sriramadasu commented on HADOOP-2759:
-------------------------------------------------

Here is a usecase : Creation of side-files by the task:
Every running job has a temporary output directory ${mapred.output.dir}/\_temporary, which is deleted at the completion of the job. And a job can be declared SUCCESSFUL before killing the speculative tasks. If the speculative task is creating a side file after the deletion of \_temporary, since _create_ api creates the parent directories, \_temporary will be created again in output directory. So, there is possibility that  \_temporary directory will be present in the output directory read by a chained job. 
More discussion is available at http://issues.apache.org/jira/browse/HADOOP-2391?focusedCommentId=12566183#action_12566183
and http://issues.apache.org/jira/browse/HADOOP-2391?focusedCommentId=12566761#action_12566761

> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>            Assignee: Pi Song
>             Fix For: 0.18.0
>
>         Attachments: hadoop-2759-complete1.patch, HADOOP-2759_1.patch, hadoop_tmp.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2759) creating a file in hdfs should not automatically create the parent directories

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567053#action_12567053 ] 

Pi Song commented on HADOOP-2759:
---------------------------------

My initial solution

1. As many other components may still refer to mkdirs (in Java sematic), let's keep it and create a new method called mkdir. The new "mkdir" will only create the specified directory *not including parents* so if all the parents don't exist, it will not allow you to create.

2. This method is implemented at the NameNode side and requires a change in ClientProtocol. As a matter of fact, if I just use ClientProtocol.exist and ClientProtocol.mkdirs in combination to implement mkdir on the client side it is possible but that incurs a bit more round-trip operations. I think I can only spend a bit more time to make it better so let's go for the better one.

3. You will have to hook up the new mkdir method in DFSClient to fix HADOOP-2391. What I do now to demonstrate the difference is just hooking up the mkdir method to  mkdir in the shell and move the old mkdir implementation (which actually is mkdirs in Java) to mkdirs

So in the shell,
hadoop dfs -mkdir  <dir>                  will not create parent dirs
hadoop dfs -mkdirs <dir>                 will create all the parents

4. In the current implementation if you mkdir a directory which already exists, there will be no error message. This has to be fixed too.



> creating a file in hdfs should not automatically create the parent directories
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-2759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2759
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-2759_1.patch
>
>
> I think it would be better if HDFS didn't automatically create directories for the user. In particular, in clean up code, it would be nice if deleting a directory couldn't be undone by mistake by a process that hasn't been killed yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.