You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Christian Kunz (JIRA)" <ji...@apache.org> on 2006/08/03 22:26:13 UTC

[jira] Created: (HADOOP-423) file paths are not normalized

file paths are not normalized
-----------------------------

                 Key: HADOOP-423
                 URL: http://issues.apache.org/jira/browse/HADOOP-423
             Project: Hadoop
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.4.0
            Reporter: Christian Kunz


Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-423) file paths are not normalized

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-423?page=comments#action_12436672 ] 
            
Doug Cutting commented on HADOOP-423:
-------------------------------------

I think, as a safeguard, we should probably also disallow "." and ".." as file or directory names at the namenode, throwing an exception for attempts to create directories or files with these names.

While we're at it, should we also prohibit "/" and ":" in names?  I think so.  Currently it is impossible to create a Path with a slash in a name, although colons are permitted.

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>         Attachments: pathnorm.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Wendy Chien updated HADOOP-423:
-------------------------------

    Attachment:     (was: pathnorm2.patch)

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>             Fix For: 0.7.0
>
>         Attachments: pathnorm3.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Wendy Chien updated HADOOP-423:
-------------------------------

    Status: Patch Available  (was: Open)

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>             Fix For: 0.7.0
>
>         Attachments: pathnorm3.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-423) file paths are not normalized

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-423?page=comments#action_12438586 ] 
            
Doug Cutting commented on HADOOP-423:
-------------------------------------

> I don't understand where you don't want to call Path.toString(), in DFSClient?

You're right.  I mistakenly assumed that src.toString() was used to convert a Path to a String, when it's really used to convert a UTF8 to a String.  Sorry!

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>             Fix For: 0.7.0
>
>         Attachments: pathnorm2.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-423?page=comments#action_12437696 ] 
            
Wendy Chien commented on HADOOP-423:
------------------------------------

I can add a check to disallow "." and ".." (and prohibit ":") as file or directory names.  Should this be done in DFSClient or the namenode?  We can save an rpc by putting it in the client.

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>         Attachments: pathnorm.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-423?page=comments#action_12438585 ] 
            
Wendy Chien commented on HADOOP-423:
------------------------------------

You're right, I've changed the logic in the namenode check.

I don't understand where you don't want to call Path.toString(), in DFSClient?  The namenode only gets a String, which I parse for the offending sequences.    Do you mean we should convert it to a Path there?

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>             Fix For: 0.7.0
>
>         Attachments: pathnorm2.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-423?page=comments#action_12433008 ] 
            
Wendy Chien commented on HADOOP-423:
------------------------------------

To fix this, we'll detect and normalize relative paths in the Path class.  For backwards compatibility, we'll allow people to rename or remove existing files/dirs that contain '.' or '..' .  Or do people think that's not necessary because it's unlikely anyone has been using '.' and '..' in their paths intentionally?  (Though I suppose if they did unintentionally, it might be nice to let them remove them.)
      

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>             Fix For: 0.6.0
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Wendy Chien updated HADOOP-423:
-------------------------------

    Attachment: pathnorm3.patch

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>             Fix For: 0.7.0
>
>         Attachments: pathnorm3.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Sameer Paranjpye updated HADOOP-423:
------------------------------------

      Component/s: dfs
                       (was: fs)
    Fix Version/s: 0.6.0
         Assignee: Wendy Chien

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>             Fix For: 0.6.0
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Doug Cutting updated HADOOP-423:
--------------------------------

           Status: Open  (was: Patch Available)
    Fix Version/s: 0.7.0

I think the check in the NameNode isn't quite right.  We should never permit "." or ".." at any point in a path here, since paths should always be fully qualifited and normalized by the time they reach the namenode, right?

Also, it seems a shame to have to call Path.toString().  Perhaps we should add a Path method like:

  public String element(int i) { return elements[i]; }

Then the path checker could be implemented with something like:

if (!path.isAbsolute()) {
  throw ... 
}
for (int i = 0; i < path.depth(); i++) {
  String element = path.element(i);
  if (element.equals("..") || element.equals(".") || element.indexOf("/") >= 0) {
    throw ...
  }
}


> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>             Fix For: 0.7.0
>
>         Attachments: pathnorm2.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Wendy Chien updated HADOOP-423:
-------------------------------

    Attachment: pathnorm.patch

The patch also includes changes in Path.java as well as added tests in TestPath.java.

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>         Attachments: pathnorm.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Wendy Chien updated HADOOP-423:
-------------------------------

    Attachment: pathnorm2.patch

This patch also contains the check in the namenode. 

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>         Attachments: pathnorm2.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Doug Cutting updated HADOOP-423:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I just committed this.  Thanks, Wendy!

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>             Fix For: 0.7.0
>
>         Attachments: pathnorm3.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-423) file paths are not normalized

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-423?page=comments#action_12437698 ] 
            
Doug Cutting commented on HADOOP-423:
-------------------------------------

I think we want it as a sanity check in the namenode, so that we're sure they never enter the filesystem.  I don't think it's worth trying to optimize this.

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>         Attachments: pathnorm.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Wendy Chien updated HADOOP-423:
-------------------------------

    Attachment:     (was: pathnorm.patch)

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>         Attachments: pathnorm2.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-423) file paths are not normalized

Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-423?page=all ]

Wendy Chien updated HADOOP-423:
-------------------------------

    Status: Patch Available  (was: Open)

> file paths are not normalized
> -----------------------------
>
>                 Key: HADOOP-423
>                 URL: http://issues.apache.org/jira/browse/HADOOP-423
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.4.0
>            Reporter: Christian Kunz
>         Assigned To: Wendy Chien
>         Attachments: pathnorm.patch
>
>
> Paths containing  '.' or '..' are not normalized -- the corresponding directories actually seem to be created. This is inconsistent and unexpected behavior, and  is particularly painful when running from a working directory, using relative paths that start with ...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira