You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jackye1995 (via GitHub)" <gi...@apache.org> on 2023/02/08 15:44:06 UTC

[GitHub] [iceberg] jackye1995 opened a new pull request, #6772: Core: enforce writing POSIX compatible paths

jackye1995 opened a new pull request, #6772:
URL: https://github.com/apache/iceberg/pull/6772

   fixes #6758 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6772: Core: enforce writing POSIX compatible paths for data location

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#discussion_r1103698178


##########
core/src/main/java/org/apache/iceberg/util/LocationUtil.java:
##########
@@ -33,4 +34,8 @@ public static String stripTrailingSlash(String path) {
     }
     return result;
   }
+
+  public static String posixNormalize(String path) {
+    return Paths.get(path).normalize().toString();

Review Comment:
   Looks like using URI is still not the right way, because URI would fail with characters like space, which are supposed to be escaped with URL encoding like `%20`
   
   I think what should be done is that, for path with scheme, everything after that should use `Paths.get(path).normalize().toString()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on pull request #6772: Core: enforce writing POSIX compatible paths

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#issuecomment-1422829117

   @RussellSpitzer @electrum @findepi @amogh-jahagirdar 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] electrum commented on a diff in pull request #6772: Core: enforce writing POSIX compatible paths

Posted by "electrum (via GitHub)" <gi...@apache.org>.
electrum commented on code in PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#discussion_r1100535487


##########
core/src/main/java/org/apache/iceberg/util/LocationUtil.java:
##########
@@ -33,4 +34,8 @@ public static String stripTrailingSlash(String path) {
     }
     return result;
   }
+
+  public static String posixNormalize(String path) {
+    return Paths.get(path).normalize().toString();

Review Comment:
   This method is not suitable for Iceberg paths, since Java `Path` is for paths, but Iceberg paths are actually URIs. It mangles URIs and doesn't understand the scheme vs path distinction:
   ```
   jshell> Paths.get("s3://foo/bar").toString()
   $2 ==> "s3:/foo/bar"
   
   jshell> Paths.get("s3://foo/bar/../..").normalize().toString()
   $5 ==> "s3:"
   
   jshell> Paths.get("s3://foo/bar/../../..").normalize().toString()
   $6 ==> ""
   ```
   
   `URI.normalize()` seems better, though it has interesting behavior where it preserves leading `..` path segments at the start of the path that would go past the root:
   
   ```
   jshell> URI.create("s3://foo/abc/../../..").normalize()
   $11 ==> s3://foo/../..
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on pull request #6772: Core: enforce writing POSIX compatible paths for data location

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#issuecomment-1426865674

   @amogh-jahagirdar @rajarshisarkar @singhpk234 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on pull request #6772: Core: enforce writing POSIX compatible paths for data location

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#issuecomment-1426873914

   We have different places for creating file path names,
   - `LocationProvider` for data files
   - `TableOperations.metadataFileLocation` for manifests and manifest lists
   -  catalog specific impl for metadata file location (e.g. `BaseMetastoreTableOperations.newTableMetadataFilePath`)
   
   So I will restrict this PR to just data location, and do other ones later to avoid having too many changes in different places.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6772: Core: enforce writing POSIX compatible paths for data location

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#discussion_r1103698178


##########
core/src/main/java/org/apache/iceberg/util/LocationUtil.java:
##########
@@ -33,4 +34,8 @@ public static String stripTrailingSlash(String path) {
     }
     return result;
   }
+
+  public static String posixNormalize(String path) {
+    return Paths.get(path).normalize().toString();

Review Comment:
   Looks like using URI is still not the right way, because URI would fail with characters like space, which are supposed to be escaped with URL encoding like `%20`. However, it is a valid posix path.
   
   I think what should be done is that, for path with scheme, everything after that should use `Paths.get(path).normalize().toString()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6772: Core: enforce writing POSIX compatible paths for data location

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#discussion_r1103698178


##########
core/src/main/java/org/apache/iceberg/util/LocationUtil.java:
##########
@@ -33,4 +34,8 @@ public static String stripTrailingSlash(String path) {
     }
     return result;
   }
+
+  public static String posixNormalize(String path) {
+    return Paths.get(path).normalize().toString();

Review Comment:
   Looks like using URI is still not the right way, because URI would fail with characters like space, which are supposed to be escaped with URL encoding like `%20`
   
   I think what should be done is that, for path with scheme and optional authority, everything after that should use `Paths.get(path).normalize().toString()`.



##########
core/src/main/java/org/apache/iceberg/util/LocationUtil.java:
##########
@@ -33,4 +34,8 @@ public static String stripTrailingSlash(String path) {
     }
     return result;
   }
+
+  public static String posixNormalize(String path) {
+    return Paths.get(path).normalize().toString();

Review Comment:
   Looks like using URI is still not the right way, because URI would fail with characters like space, which are supposed to be escaped with URL encoding like `%20`
   
   I think what should be done is that, for path with scheme and authority, everything after that should use `Paths.get(path).normalize().toString()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: enforce writing POSIX compatible paths for data location [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#issuecomment-2323082393

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: enforce writing POSIX compatible paths for data location [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#issuecomment-2308585052

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6772: Core: enforce writing POSIX compatible paths

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#discussion_r1100330049


##########
core/src/test/java/org/apache/iceberg/TestLocationProvider.java:
##########
@@ -297,4 +297,17 @@ public void testObjectStorageWithinTableLocation() {
     Assert.assertEquals(
         "Fourth part should be the file name passed in", "test.parquet", parts.get(3));
   }
+
+  @Test

Review Comment:
   Put up for discussion first while writing tests in case we want to talk about it during community sync



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6772: Core: enforce writing POSIX compatible paths for data location

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#discussion_r1103706868


##########
core/src/test/java/org/apache/iceberg/util/TestLocationUtil.java:
##########
@@ -61,4 +61,124 @@ public void testStripTrailingSlashWithInvalidPath() {
           () -> LocationUtil.stripTrailingSlash(invalidPath));
     }
   }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityS3Style() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "s3://bucket/dir",
+        LocationUtil.posixNormalize("s3://bucket/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "s3://bucket/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityHdfsStyle() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "hdfs://1.2.3.4/dir",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "hdfs://1.2.3.4/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndWithoutAuthority() {
+    Assert.assertEquals(
+        "Must work with the root directory representation",
+        "file:///",

Review Comment:
   This is a place where Hadoop Path behavior differs, it resolves to `file:/`, but that seems like an invalid resolution.
   
   Also I am treating `/` as a special case, both here and in a path without a scheme. For paths with a scheme and authority, we can use `scheme://authority` as the root path without ambiguity. But `file://` (just scheme) and ` ` (empty path) do not mean root location, only `file:///` and `/` mean that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6772: Core: enforce writing POSIX compatible paths for data location

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#discussion_r1103706868


##########
core/src/test/java/org/apache/iceberg/util/TestLocationUtil.java:
##########
@@ -61,4 +61,124 @@ public void testStripTrailingSlashWithInvalidPath() {
           () -> LocationUtil.stripTrailingSlash(invalidPath));
     }
   }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityS3Style() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "s3://bucket/dir",
+        LocationUtil.posixNormalize("s3://bucket/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "s3://bucket/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityHdfsStyle() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "hdfs://1.2.3.4/dir",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "hdfs://1.2.3.4/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndWithoutAuthority() {
+    Assert.assertEquals(
+        "Must work with the root directory representation",
+        "file:///",

Review Comment:
   This is a place where Hadoop Path differs, it resolves to `file:/`, but that seems like an invalid resolution.
   
   Also I am treating `/` as a special case, both here and in a path without a scheme. For paths with a scheme and authority, we can use `scheme://authority` as the root path without ambiguity. But `file://` (just scheme) and `` (empty path) does not mean root location, only `file:///` and `/` mean that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: enforce writing POSIX compatible paths for data location [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6772: Core: enforce writing POSIX compatible paths for data location
URL: https://github.com/apache/iceberg/pull/6772


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6772: Core: enforce writing POSIX compatible paths for data location

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#discussion_r1103706868


##########
core/src/test/java/org/apache/iceberg/util/TestLocationUtil.java:
##########
@@ -61,4 +61,124 @@ public void testStripTrailingSlashWithInvalidPath() {
           () -> LocationUtil.stripTrailingSlash(invalidPath));
     }
   }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityS3Style() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "s3://bucket/dir",
+        LocationUtil.posixNormalize("s3://bucket/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "s3://bucket/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityHdfsStyle() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "hdfs://1.2.3.4/dir",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "hdfs://1.2.3.4/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndWithoutAuthority() {
+    Assert.assertEquals(
+        "Must work with the root directory representation",
+        "file:///",

Review Comment:
   This is a place where Hadoop Path differs, it resolves to `file:/`, but that seems like an invalid resolution.
   
   Also I am treating `/` as a special case, both here and in a path without a scheme. If we don't do this, then a user defining a table root location `/` will be converted to empty path and all locations become relative.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6772: Core: enforce writing POSIX compatible paths for data location

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #6772:
URL: https://github.com/apache/iceberg/pull/6772#discussion_r1103706868


##########
core/src/test/java/org/apache/iceberg/util/TestLocationUtil.java:
##########
@@ -61,4 +61,124 @@ public void testStripTrailingSlashWithInvalidPath() {
           () -> LocationUtil.stripTrailingSlash(invalidPath));
     }
   }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityS3Style() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "s3://bucket/dir",
+        LocationUtil.posixNormalize("s3://bucket/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "s3://bucket/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityHdfsStyle() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "hdfs://1.2.3.4/dir",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "hdfs://1.2.3.4/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndWithoutAuthority() {
+    Assert.assertEquals(
+        "Must work with the root directory representation",
+        "file:///",

Review Comment:
   This is a place where Hadoop Path differs, it resolves to `file:/`, but that seems like an invalid resolution.
   
   Also I am treating `/` as a special case, both here and in a path without a scheme. For paths with a scheme and authority, we can use `scheme://authority` as the root path without ambiguity. But `file://` (just scheme) and ` ` (empty path) does not mean root location, only `file:///` and `/` mean that.



##########
core/src/test/java/org/apache/iceberg/util/TestLocationUtil.java:
##########
@@ -61,4 +61,124 @@ public void testStripTrailingSlashWithInvalidPath() {
           () -> LocationUtil.stripTrailingSlash(invalidPath));
     }
   }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityS3Style() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "s3://bucket",
+        LocationUtil.posixNormalize("s3://bucket/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "s3://bucket/dir",
+        LocationUtil.posixNormalize("s3://bucket/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "s3://bucket/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "s3://bucket/foo/bar",
+        LocationUtil.posixNormalize("s3://bucket/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndAuthorityHdfsStyle() {
+    Assert.assertEquals(
+        "Must work with a root authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4"));
+    Assert.assertEquals(
+        "Must remove / from the end of authority",
+        "hdfs://1.2.3.4",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/"));
+    Assert.assertEquals(
+        "Must remove / from the end of directory",
+        "hdfs://1.2.3.4/dir",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/dir/"));
+    Assert.assertEquals(
+        "Must change // to / to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo//bar"));
+    Assert.assertEquals(
+        "Must resolve .. to previous directory to enforce posix path",
+        "hdfs://1.2.3.4/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/../bar"));
+    Assert.assertEquals(
+        "Must resolve . to current directory to enforce posix path",
+        "hdfs://1.2.3.4/foo/bar",
+        LocationUtil.posixNormalize("hdfs://1.2.3.4/foo/.//bar"));
+  }
+
+  @Test
+  public void testPosixNormalizePathsWithSchemeAndWithoutAuthority() {
+    Assert.assertEquals(
+        "Must work with the root directory representation",
+        "file:///",

Review Comment:
   This is a place where Hadoop Path differs, it resolves to `file:/`, but that seems like an invalid resolution.
   
   Also I am treating `/` as a special case, both here and in a path without a scheme. For paths with a scheme and authority, we can use `scheme://authority` as the root path without ambiguity. But `file://` (just scheme) and ` ` (empty path) do not mean root location, only `file:///` and `/` mean that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org