You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/07/10 23:00:06 UTC

[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #1196: Keyput

bharatviswa504 opened a new pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196


   ## What changes were proposed in this pull request?
   
   (Please fill in changes proposed in this fix)
   
   ## What is the link to the Apache JIRA
   
   (Please create an issue in ASF JIRA before opening a pull request,
   and you need to set the title of the pull request which starts with
   the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
   
   Please replace this section with the link to the Apache JIRA)
   
   ## How was this patch tested?
   
   (Please explain how this patch was tested. Ex: unit tests, manual tests)
   (If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456660788



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -265,4 +273,72 @@ public AuditMessage buildAuditMessage(AuditAction op,
     auditMap.put(OzoneConsts.VOLUME, volume);
     return auditMap;
   }
+
+
+  public static String validateAndNormalizeKey(boolean enableFileSystemPaths,
+      String keyName) throws OMException {
+    if (enableFileSystemPaths) {
+      return validateAndNormalizeKey(keyName);
+    } else {
+      return keyName;
+    }
+  }
+
+  @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+  public static String validateAndNormalizeKey(String keyName)
+      throws OMException {
+    String normalizedKeyName;
+    if (keyName.startsWith(OM_KEY_PREFIX)) {
+      normalizedKeyName = Paths.get(keyName).toUri().normalize().getPath();
+    } else {
+      normalizedKeyName = Paths.get(OM_KEY_PREFIX, keyName).toUri()
+          .normalize().getPath();
+    }
+    if (!keyName.equals(normalizedKeyName)) {
+      LOG.debug("Normalized key {} to {} ", keyName,
+          normalizedKeyName.substring(1));
+    }
+    return isValidKeyPath(normalizedKeyName.substring(1));
+  }
+
+  /**
+   * Whether the pathname is valid.  Check key names which contain a
+   * ":", ".", "..", "//", "". If it has any of these characters throws
+   * OMException, else return the path.
+   */
+  private static String isValidKeyPath(String path) throws OMException {
+    boolean isValid = true;
+
+    // If keyName is empty string throw error.
+    if (path.length() == 0) {
+      throw new OMException("Invalid KeyPath, empty keyName" + path,
+          INVALID_KEY_NAME);
+    } else if(path.startsWith("/")) {
+      isValid = false;
+    } else {
+      // Check for ".." "." ":" "/"
+      String[] components = StringUtils.split(path, '/');
+      for (int i = 0; i < components.length; i++) {
+        String element = components[i];
+        if (element.equals(".") ||
+            (element.contains(":")) ||
+            (element.contains("/") || element.equals(".."))) {
+          isValid = false;
+          break;
+        }
+
+        // The string may end with a /, but not have

Review comment:
       Yes, we don't allow for a keyName with trailing /. This method can return after normalization in some cases. So, we checked in OmKeyCreateRequest




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456631086



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -327,7 +344,164 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+
+    keyName = "dir1/dir2/dir3/file1";
+    createAndCheck(keyName);
+
+    // Key with leading '/'.
+    String keyName = "/a/b/c/file1";
+    createAndCheck(keyName);
+
+    // Commit openKey entry.
+    TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+        keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+    // Now create another file in same dir path.
+    keyName = "/a/b/c/file2";
+    createAndCheck(keyName);
+
+    // Create key with multiple /'s
+    // converted to a/b/c/file5
+    keyName = "///a/b///c///file5";
+    createAndCheck(keyName);
+
+    // converted to a/b/c/.../file3
+    keyName = "///a/b///c//.../file3";
+    createAndCheck(keyName);
+
+    // converted to r1/r2
+    keyName = "././r1/r2/";
+    createAndCheck(keyName);
+
+    // converted to ..d1/d2/d3
+    keyName = "..d1/d2/d3/";
+    createAndCheck(keyName);
+
+    // Create a file, where a file already exists in the path.
+    // Now try with a file exists in path. Should fail.
+    keyName = "/a/b/c/file1/file3";
+    checkNotAFile(keyName);
+
+    // Empty keyName.
+    keyName = "";
+    checkNotAValidPath(keyName);
+
+    // Key name ends with /
+    keyName = "/a/./";
+    checkNotAValidPath(keyName);
+
+    keyName = "/////";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b/c";

Review comment:
       Also let's add a check for `../a/b` and `/../a/b`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456666947



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -327,7 +344,164 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+
+    keyName = "dir1/dir2/dir3/file1";
+    createAndCheck(keyName);
+
+    // Key with leading '/'.
+    String keyName = "/a/b/c/file1";
+    createAndCheck(keyName);
+
+    // Commit openKey entry.
+    TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+        keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+    // Now create another file in same dir path.
+    keyName = "/a/b/c/file2";
+    createAndCheck(keyName);
+
+    // Create key with multiple /'s
+    // converted to a/b/c/file5
+    keyName = "///a/b///c///file5";
+    createAndCheck(keyName);
+
+    // converted to a/b/c/.../file3
+    keyName = "///a/b///c//.../file3";
+    createAndCheck(keyName);
+
+    // converted to r1/r2
+    keyName = "././r1/r2/";
+    createAndCheck(keyName);
+
+    // converted to ..d1/d2/d3

Review comment:
       we will fail, even after normalize if we have a case like /a/. -> a/




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r453761889



##########
File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
##########
@@ -239,4 +239,11 @@ private OMConfigKeys() {
           "ozone.om.keyname.character.check.enabled";
   public static final boolean OZONE_OM_KEYNAME_CHARACTER_CHECK_ENABLED_DEFAULT =
           false;
+
+  // This config needs to be enabled, when S3G created objects will be used
+  // FileSystem.

Review comment:
       used via FileSystem API (reword comment).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r453761889



##########
File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
##########
@@ -239,4 +239,11 @@ private OMConfigKeys() {
           "ozone.om.keyname.character.check.enabled";
   public static final boolean OZONE_OM_KEYNAME_CHARACTER_CHECK_ENABLED_DEFAULT =
           false;
+
+  // This config needs to be enabled, when S3G created objects will be used
+  // FileSystem.

Review comment:
       used via FileSystem API.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
arp7 commented on pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-657658582


   Can you update the PR template a bit to describe the high-level approach (was fix made in OM or S3G)? 🙂 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-660250249


   Rebased with latest apache main branch.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456630734



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -327,7 +344,164 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+
+    keyName = "dir1/dir2/dir3/file1";
+    createAndCheck(keyName);
+
+    // Key with leading '/'.
+    String keyName = "/a/b/c/file1";
+    createAndCheck(keyName);
+
+    // Commit openKey entry.
+    TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+        keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+    // Now create another file in same dir path.
+    keyName = "/a/b/c/file2";
+    createAndCheck(keyName);
+
+    // Create key with multiple /'s
+    // converted to a/b/c/file5
+    keyName = "///a/b///c///file5";
+    createAndCheck(keyName);
+
+    // converted to a/b/c/.../file3
+    keyName = "///a/b///c//.../file3";
+    createAndCheck(keyName);
+
+    // converted to r1/r2
+    keyName = "././r1/r2/";
+    createAndCheck(keyName);
+
+    // converted to ..d1/d2/d3

Review comment:
       Why is this converted to `../d1/d2/d3`? Should it be failed instead due to trailing `/`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455968767



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##########
@@ -79,15 +79,11 @@ public static OMPathInfo verifyFilesInPath(
       String dbDirKeyName = omMetadataManager.getOzoneDirKey(volumeName,
           bucketName, pathName);
 
-      if (omMetadataManager.getKeyTable().isExist(dbKeyName)) {
-        // Found a file in the given path.
-        // Check if this is actual file or a file in the given path
-        if (dbKeyName.equals(fileNameFromDetails)) {
-          result = OMDirectoryResult.FILE_EXISTS;
-        } else {
-          result = OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
-        }
-      } else if (omMetadataManager.getKeyTable().isExist(dbDirKeyName)) {
+      // Check first for dir. This is to handle leading "/" in path, which

Review comment:
       Yes. This change was required, when we are allowing leading / in keyName. I will revert the change




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456625982



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
##########
@@ -91,6 +99,19 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     if(checkKeyNameEnabled){
       OmUtils.validateKeyName(keyArgs.getKeyName());
     }
+
+    String keyPath = keyArgs.getKeyName();
+    if (ozoneManager.getEnableFileSystemPaths()) {
+      // If enabled, disallow keys with trailing /. As in fs semantics
+      // directories end with trailing /.
+      keyPath = validateAndNormalizeKey(
+          ozoneManager.getEnableFileSystemPaths(), keyPath);
+      if (keyPath.endsWith("/")) {
+        throw new OMException("Invalid KeyPath: " + keyPath,
+            OMException.ResultCodes.INVALID_KEY_NAME);

Review comment:
       Let's also mention why it is invalid, else user may be scratching their head. 🙂 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455410736



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.ozone.om.request;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.ozone.om.request.OMClientRequest.getNormalizedKey;
+
+/**
+ * Class to test normalize paths.
+ */
+public class TestNormalizePaths {
+
+  @Test
+  public void testNormalizePathsEnabled() {
+
+    Assert.assertEquals("a/b/c/d",
+        getNormalizedKey(true, "a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        getNormalizedKey(true, "/a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        getNormalizedKey(true, "////a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        getNormalizedKey(true, "////a/b/////c/d"));
+    Assert.assertEquals("a/b/c/...../d",
+        getNormalizedKey(true, "////a/b/////c/...../d"));
+    Assert.assertEquals("a/b/d",
+        getNormalizedKey(true, "/a/b/c/../d"));
+    Assert.assertEquals("a/d",

Review comment:
       Done

##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -265,4 +272,24 @@ public AuditMessage buildAuditMessage(AuditAction op,
     auditMap.put(OzoneConsts.VOLUME, volume);
     return auditMap;
   }
+
+  @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+  public static String getNormalizedKey(boolean enableFileSystemPaths,

Review comment:
       Moved common logic to a new method.

##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -265,4 +272,24 @@ public AuditMessage buildAuditMessage(AuditAction op,
     auditMap.put(OzoneConsts.VOLUME, volume);
     return auditMap;
   }
+
+  @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+  public static String getNormalizedKey(boolean enableFileSystemPaths,
+      String keyName) {
+    if (enableFileSystemPaths) {
+      String normalizedKeyName;
+      if (keyName.startsWith(OM_KEY_PREFIX)) {
+        normalizedKeyName = Paths.get(keyName).toUri().normalize().getPath();
+      } else {
+        normalizedKeyName = Paths.get(OM_KEY_PREFIX, keyName).toUri()
+            .normalize().getPath();
+      }
+      if (!keyName.equals(normalizedKeyName)) {
+        LOG.debug("Normalized key {} to {} ", keyName, normalizedKeyName);

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456631870



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -327,7 +344,164 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+
+    keyName = "dir1/dir2/dir3/file1";
+    createAndCheck(keyName);
+
+    // Key with leading '/'.
+    String keyName = "/a/b/c/file1";
+    createAndCheck(keyName);
+
+    // Commit openKey entry.
+    TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+        keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+    // Now create another file in same dir path.
+    keyName = "/a/b/c/file2";
+    createAndCheck(keyName);
+
+    // Create key with multiple /'s
+    // converted to a/b/c/file5
+    keyName = "///a/b///c///file5";
+    createAndCheck(keyName);
+
+    // converted to a/b/c/.../file3
+    keyName = "///a/b///c//.../file3";
+    createAndCheck(keyName);
+
+    // converted to r1/r2
+    keyName = "././r1/r2/";
+    createAndCheck(keyName);
+
+    // converted to ..d1/d2/d3
+    keyName = "..d1/d2/d3/";
+    createAndCheck(keyName);
+
+    // Create a file, where a file already exists in the path.
+    // Now try with a file exists in path. Should fail.
+    keyName = "/a/b/c/file1/file3";
+    checkNotAFile(keyName);
+
+    // Empty keyName.
+    keyName = "";
+    checkNotAValidPath(keyName);
+
+    // Key name ends with /
+    keyName = "/a/./";
+    checkNotAValidPath(keyName);
+
+    keyName = "/////";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b/c";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b/c/";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b:/c/";
+    checkNotAValidPath(keyName);
+
+    keyName = ":/c/";
+    checkNotAValidPath(keyName);
+
+    keyName = "";
+    checkNotAValidPath(keyName);
+
+  }
+
 
+  private void checkNotAValidPath(String keyName) {
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    try {
+      omKeyCreateRequest.preExecute(ozoneManager);
+      fail("checkNotAValidPath failed for path" + keyName);
+    } catch (IOException ex) {
+      Assert.assertTrue(ex instanceof OMException);
+      OMException omException = (OMException) ex;
+      Assert.assertEquals(OMException.ResultCodes.INVALID_KEY_NAME,
+          omException.getResult());
+    }
+
+
+  }
+  private void checkNotAFile(String keyName) throws Exception {
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+    omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    OMClientResponse omClientResponse =
+        omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+            101L, ozoneManagerDoubleBufferHelper);
+
+    Assert.assertEquals(NOT_A_FILE,
+        omClientResponse.getOMResponse().getStatus());
+  }
+
+
+  private void createAndCheck(String keyName) throws Exception {
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+    omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    OMClientResponse omClientResponse =
+        omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+            101L, ozoneManagerDoubleBufferHelper);
+
+    Assert.assertEquals(OK, omClientResponse.getOMResponse().getStatus());
+
+    checkCreatedPaths(omKeyCreateRequest, omRequest, keyName);
+  }
+
+  private void checkCreatedPaths(OMKeyCreateRequest omKeyCreateRequest,
+      OMRequest omRequest, String keyName) throws Exception {
+    keyName = omKeyCreateRequest.validateAndNormalizeKey(true, keyName);

Review comment:
       Let's add a separate test case for `validateAndNormalizeKey` itself. All it does is call `validateAndNormalizeKey` with bunch of hard-coded input and asserts on the expected output.

##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -327,7 +344,164 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+
+    keyName = "dir1/dir2/dir3/file1";
+    createAndCheck(keyName);
+
+    // Key with leading '/'.
+    String keyName = "/a/b/c/file1";
+    createAndCheck(keyName);
+
+    // Commit openKey entry.
+    TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+        keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+    // Now create another file in same dir path.
+    keyName = "/a/b/c/file2";
+    createAndCheck(keyName);
+
+    // Create key with multiple /'s
+    // converted to a/b/c/file5
+    keyName = "///a/b///c///file5";
+    createAndCheck(keyName);
+
+    // converted to a/b/c/.../file3
+    keyName = "///a/b///c//.../file3";
+    createAndCheck(keyName);
+
+    // converted to r1/r2
+    keyName = "././r1/r2/";
+    createAndCheck(keyName);
+
+    // converted to ..d1/d2/d3
+    keyName = "..d1/d2/d3/";
+    createAndCheck(keyName);
+
+    // Create a file, where a file already exists in the path.
+    // Now try with a file exists in path. Should fail.
+    keyName = "/a/b/c/file1/file3";
+    checkNotAFile(keyName);
+
+    // Empty keyName.
+    keyName = "";
+    checkNotAValidPath(keyName);
+
+    // Key name ends with /
+    keyName = "/a/./";
+    checkNotAValidPath(keyName);
+
+    keyName = "/////";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b/c";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b/c/";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b:/c/";
+    checkNotAValidPath(keyName);
+
+    keyName = ":/c/";
+    checkNotAValidPath(keyName);
+
+    keyName = "";
+    checkNotAValidPath(keyName);
+
+  }
+
 
+  private void checkNotAValidPath(String keyName) {
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    try {
+      omKeyCreateRequest.preExecute(ozoneManager);
+      fail("checkNotAValidPath failed for path" + keyName);
+    } catch (IOException ex) {
+      Assert.assertTrue(ex instanceof OMException);
+      OMException omException = (OMException) ex;
+      Assert.assertEquals(OMException.ResultCodes.INVALID_KEY_NAME,
+          omException.getResult());
+    }
+
+
+  }
+  private void checkNotAFile(String keyName) throws Exception {
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+    omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    OMClientResponse omClientResponse =
+        omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+            101L, ozoneManagerDoubleBufferHelper);
+
+    Assert.assertEquals(NOT_A_FILE,
+        omClientResponse.getOMResponse().getStatus());
+  }
+
+
+  private void createAndCheck(String keyName) throws Exception {
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+    omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    OMClientResponse omClientResponse =
+        omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+            101L, ozoneManagerDoubleBufferHelper);
+
+    Assert.assertEquals(OK, omClientResponse.getOMResponse().getStatus());
+
+    checkCreatedPaths(omKeyCreateRequest, omRequest, keyName);
+  }
+
+  private void checkCreatedPaths(OMKeyCreateRequest omKeyCreateRequest,
+      OMRequest omRequest, String keyName) throws Exception {
+    keyName = omKeyCreateRequest.validateAndNormalizeKey(true, keyName);
+    // Check intermediate directories created or not.
+    Path keyPath = Paths.get(keyName);
+    checkIntermediatePaths(keyPath);

Review comment:
       Nice!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456666760



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -327,7 +344,164 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+
+    keyName = "dir1/dir2/dir3/file1";
+    createAndCheck(keyName);
+
+    // Key with leading '/'.
+    String keyName = "/a/b/c/file1";
+    createAndCheck(keyName);
+
+    // Commit openKey entry.
+    TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+        keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+    // Now create another file in same dir path.
+    keyName = "/a/b/c/file2";
+    createAndCheck(keyName);
+
+    // Create key with multiple /'s
+    // converted to a/b/c/file5
+    keyName = "///a/b///c///file5";
+    createAndCheck(keyName);
+
+    // converted to a/b/c/.../file3
+    keyName = "///a/b///c//.../file3";
+    createAndCheck(keyName);
+
+    // converted to r1/r2
+    keyName = "././r1/r2/";
+    createAndCheck(keyName);
+
+    // converted to ..d1/d2/d3

Review comment:
       ..d1/d2/d3/ -> ../d1/d2/d3 it removed trailing /. Normalize will take care of removing additional /




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456625606



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -265,4 +273,72 @@ public AuditMessage buildAuditMessage(AuditAction op,
     auditMap.put(OzoneConsts.VOLUME, volume);
     return auditMap;
   }
+
+
+  public static String validateAndNormalizeKey(boolean enableFileSystemPaths,
+      String keyName) throws OMException {
+    if (enableFileSystemPaths) {
+      return validateAndNormalizeKey(keyName);
+    } else {
+      return keyName;
+    }
+  }
+
+  @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+  public static String validateAndNormalizeKey(String keyName)
+      throws OMException {
+    String normalizedKeyName;
+    if (keyName.startsWith(OM_KEY_PREFIX)) {
+      normalizedKeyName = Paths.get(keyName).toUri().normalize().getPath();
+    } else {
+      normalizedKeyName = Paths.get(OM_KEY_PREFIX, keyName).toUri()
+          .normalize().getPath();
+    }
+    if (!keyName.equals(normalizedKeyName)) {
+      LOG.debug("Normalized key {} to {} ", keyName,
+          normalizedKeyName.substring(1));
+    }
+    return isValidKeyPath(normalizedKeyName.substring(1));
+  }
+
+  /**
+   * Whether the pathname is valid.  Check key names which contain a
+   * ":", ".", "..", "//", "". If it has any of these characters throws
+   * OMException, else return the path.
+   */
+  private static String isValidKeyPath(String path) throws OMException {
+    boolean isValid = true;
+
+    // If keyName is empty string throw error.
+    if (path.length() == 0) {
+      throw new OMException("Invalid KeyPath, empty keyName" + path,
+          INVALID_KEY_NAME);
+    } else if(path.startsWith("/")) {
+      isValid = false;
+    } else {
+      // Check for ".." "." ":" "/"
+      String[] components = StringUtils.split(path, '/');
+      for (int i = 0; i < components.length; i++) {
+        String element = components[i];
+        if (element.equals(".") ||
+            (element.contains(":")) ||
+            (element.contains("/") || element.equals(".."))) {
+          isValid = false;
+          break;
+        }
+
+        // The string may end with a /, but not have

Review comment:
       I realized this is checked later in `OmKeyCreateRequest`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455961310



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##########
@@ -79,15 +79,11 @@ public static OMPathInfo verifyFilesInPath(
       String dbDirKeyName = omMetadataManager.getOzoneDirKey(volumeName,
           bucketName, pathName);
 
-      if (omMetadataManager.getKeyTable().isExist(dbKeyName)) {

Review comment:
       I am a little bit worried about changes to OMFileRequest. There is a risk of changing the behavior and invalidating the app-compat testing we have done so far.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-659696446


   Thank You @arp7 and @avijayanhwx for the review.
   I have addressed review comments.
   
   This PR only modifies behavior of KeyCreate when ozone.om.enable.filesystem.paths is enabled. (Not changed behavior of actual KeyCreate)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r453775722



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
##########
@@ -221,8 +233,45 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
       OmBucketInfo bucketInfo = omMetadataManager.getBucketTable().get(
           omMetadataManager.getBucketKey(volumeName, bucketName));
 
+      boolean createIntermediateDir =
+          ozoneManager.getConfiguration().getBoolean(
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY,
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY_DEFAULT);
+
+      // If FILE_EXISTS we just override like how we used to do for Key Create.
+      List< OzoneAcl > inheritAcls;
+      if (createIntermediateDir) {
+        OMFileRequest.OMPathInfo pathInfo =
+            OMFileRequest.verifyFilesInPath(omMetadataManager, volumeName,
+                bucketName, keyName, Paths.get(keyName));
+        OMFileRequest.OMDirectoryResult omDirectoryResult =
+            pathInfo.getDirectoryResult();
+        inheritAcls = pathInfo.getAcls();
+
+        // Check if a file or directory exists with same key name.
+        if (omDirectoryResult == DIRECTORY_EXISTS) {
+          throw new OMException("Can not write to directory: " + keyName,
+              NOT_A_FILE);
+        } else
+          if (omDirectoryResult == FILE_EXISTS_IN_GIVENPATH) {
+            throw new OMException("Can not create file: " + keyName +
+                " as there is already file in the given path", NOT_A_FILE);
+          }
+
+        missingParentInfos = OMDirectoryCreateRequest
+            .getAllParentInfo(ozoneManager, keyArgs,
+                pathInfo.getMissingParents(), inheritAcls, trxnLogIndex);
+
+        // Add cache entries for the prefix directories.

Review comment:
       @bharatviswa504 , one q. We are adding cache entries right away instead of at the end on success. Could this be a potential issue on the failure path?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456625155



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##########
@@ -141,7 +141,6 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
               ozoneManager.getPreallocateBlocksMax(),
               ozoneManager.isGrpcBlockTokenEnabled(),
               ozoneManager.getOMNodeId());
-

Review comment:
       Spurious change?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r453779672



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -328,6 +342,73 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithIntermediateDir() throws Exception {
+
+    String keyName = "a/b/c/file1";
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+    omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+    OMClientResponse omClientResponse =
+        omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+        100L, ozoneManagerDoubleBufferHelper);
+
+    Assert.assertEquals(omClientResponse.getOMResponse().getStatus(), OK);
+
+    Path keyPath = Paths.get(keyName);
+
+    // Check intermediate paths are created
+    keyPath = keyPath.getParent();
+    while(keyPath != null) {
+      Assert.assertNotNull(omMetadataManager.getKeyTable().get(

Review comment:
       Can we change this to fs.getFileStatus(keypath) so that returns status.isDir() ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r453774238



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
##########
@@ -221,8 +233,45 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
       OmBucketInfo bucketInfo = omMetadataManager.getBucketTable().get(
           omMetadataManager.getBucketKey(volumeName, bucketName));
 
+      boolean createIntermediateDir =
+          ozoneManager.getConfiguration().getBoolean(
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY,
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY_DEFAULT);
+
+      // If FILE_EXISTS we just override like how we used to do for Key Create.
+      List< OzoneAcl > inheritAcls;
+      if (createIntermediateDir) {
+        OMFileRequest.OMPathInfo pathInfo =
+            OMFileRequest.verifyFilesInPath(omMetadataManager, volumeName,
+                bucketName, keyName, Paths.get(keyName));
+        OMFileRequest.OMDirectoryResult omDirectoryResult =
+            pathInfo.getDirectoryResult();
+        inheritAcls = pathInfo.getAcls();
+
+        // Check if a file or directory exists with same key name.
+        if (omDirectoryResult == DIRECTORY_EXISTS) {
+          throw new OMException("Can not write to directory: " + keyName,

Review comment:
       Let's make the exception message slightly more descriptive to explain what is really going on. We can mention that createIntermediateDirs behavior is enabled and hence `/` has special interpretation.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r453775722



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
##########
@@ -221,8 +233,45 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
       OmBucketInfo bucketInfo = omMetadataManager.getBucketTable().get(
           omMetadataManager.getBucketKey(volumeName, bucketName));
 
+      boolean createIntermediateDir =
+          ozoneManager.getConfiguration().getBoolean(
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY,
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY_DEFAULT);
+
+      // If FILE_EXISTS we just override like how we used to do for Key Create.
+      List< OzoneAcl > inheritAcls;
+      if (createIntermediateDir) {
+        OMFileRequest.OMPathInfo pathInfo =
+            OMFileRequest.verifyFilesInPath(omMetadataManager, volumeName,
+                bucketName, keyName, Paths.get(keyName));
+        OMFileRequest.OMDirectoryResult omDirectoryResult =
+            pathInfo.getDirectoryResult();
+        inheritAcls = pathInfo.getAcls();
+
+        // Check if a file or directory exists with same key name.
+        if (omDirectoryResult == DIRECTORY_EXISTS) {
+          throw new OMException("Can not write to directory: " + keyName,
+              NOT_A_FILE);
+        } else
+          if (omDirectoryResult == FILE_EXISTS_IN_GIVENPATH) {
+            throw new OMException("Can not create file: " + keyName +
+                " as there is already file in the given path", NOT_A_FILE);
+          }
+
+        missingParentInfos = OMDirectoryCreateRequest
+            .getAllParentInfo(ozoneManager, keyArgs,
+                pathInfo.getMissingParents(), inheritAcls, trxnLogIndex);
+
+        // Add cache entries for the prefix directories.

Review comment:
       @bharatviswa504 , one q. We are adding intermediate cache entries right away instead of at the end on success. Could this be a potential issue on the failure path?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456633185



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -327,7 +344,164 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+
+    keyName = "dir1/dir2/dir3/file1";
+    createAndCheck(keyName);
+
+    // Key with leading '/'.
+    String keyName = "/a/b/c/file1";
+    createAndCheck(keyName);
+
+    // Commit openKey entry.
+    TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+        keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+    // Now create another file in same dir path.
+    keyName = "/a/b/c/file2";
+    createAndCheck(keyName);
+
+    // Create key with multiple /'s
+    // converted to a/b/c/file5
+    keyName = "///a/b///c///file5";
+    createAndCheck(keyName);
+
+    // converted to a/b/c/.../file3
+    keyName = "///a/b///c//.../file3";
+    createAndCheck(keyName);
+
+    // converted to r1/r2
+    keyName = "././r1/r2/";
+    createAndCheck(keyName);
+
+    // converted to ..d1/d2/d3
+    keyName = "..d1/d2/d3/";
+    createAndCheck(keyName);
+
+    // Create a file, where a file already exists in the path.
+    // Now try with a file exists in path. Should fail.
+    keyName = "/a/b/c/file1/file3";
+    checkNotAFile(keyName);
+
+    // Empty keyName.
+    keyName = "";
+    checkNotAValidPath(keyName);
+
+    // Key name ends with /
+    keyName = "/a/./";
+    checkNotAValidPath(keyName);
+
+    keyName = "/////";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b/c";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b/c/";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b:/c/";
+    checkNotAValidPath(keyName);
+
+    keyName = ":/c/";
+    checkNotAValidPath(keyName);
+
+    keyName = "";
+    checkNotAValidPath(keyName);
+
+  }
+
 
+  private void checkNotAValidPath(String keyName) {
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    try {
+      omKeyCreateRequest.preExecute(ozoneManager);
+      fail("checkNotAValidPath failed for path" + keyName);
+    } catch (IOException ex) {
+      Assert.assertTrue(ex instanceof OMException);
+      OMException omException = (OMException) ex;
+      Assert.assertEquals(OMException.ResultCodes.INVALID_KEY_NAME,
+          omException.getResult());
+    }
+
+
+  }
+  private void checkNotAFile(String keyName) throws Exception {
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+    omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    OMClientResponse omClientResponse =
+        omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+            101L, ozoneManagerDoubleBufferHelper);
+
+    Assert.assertEquals(NOT_A_FILE,
+        omClientResponse.getOMResponse().getStatus());
+  }
+
+
+  private void createAndCheck(String keyName) throws Exception {
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+    omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    OMClientResponse omClientResponse =
+        omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+            101L, ozoneManagerDoubleBufferHelper);
+
+    Assert.assertEquals(OK, omClientResponse.getOMResponse().getStatus());
+
+    checkCreatedPaths(omKeyCreateRequest, omRequest, keyName);
+  }
+
+  private void checkCreatedPaths(OMKeyCreateRequest omKeyCreateRequest,
+      OMRequest omRequest, String keyName) throws Exception {
+    keyName = omKeyCreateRequest.validateAndNormalizeKey(true, keyName);

Review comment:
       Ignore this comment, I see it above now. Was reviewing files out of order.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455960385



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##########
@@ -79,15 +79,11 @@ public static OMPathInfo verifyFilesInPath(
       String dbDirKeyName = omMetadataManager.getOzoneDirKey(volumeName,
           bucketName, pathName);
 
-      if (omMetadataManager.getKeyTable().isExist(dbKeyName)) {
-        // Found a file in the given path.
-        // Check if this is actual file or a file in the given path
-        if (dbKeyName.equals(fileNameFromDetails)) {
-          result = OMDirectoryResult.FILE_EXISTS;
-        } else {
-          result = OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
-        }
-      } else if (omMetadataManager.getKeyTable().isExist(dbDirKeyName)) {
+      // Check first for dir. This is to handle leading "/" in path, which

Review comment:
       Leading `/` in path should be dropped at this point, right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456622410



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -265,4 +273,72 @@ public AuditMessage buildAuditMessage(AuditAction op,
     auditMap.put(OzoneConsts.VOLUME, volume);
     return auditMap;
   }
+
+
+  public static String validateAndNormalizeKey(boolean enableFileSystemPaths,
+      String keyName) throws OMException {
+    if (enableFileSystemPaths) {
+      return validateAndNormalizeKey(keyName);
+    } else {
+      return keyName;
+    }
+  }
+
+  @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+  public static String validateAndNormalizeKey(String keyName)
+      throws OMException {
+    String normalizedKeyName;
+    if (keyName.startsWith(OM_KEY_PREFIX)) {
+      normalizedKeyName = Paths.get(keyName).toUri().normalize().getPath();
+    } else {
+      normalizedKeyName = Paths.get(OM_KEY_PREFIX, keyName).toUri()
+          .normalize().getPath();
+    }
+    if (!keyName.equals(normalizedKeyName)) {
+      LOG.debug("Normalized key {} to {} ", keyName,
+          normalizedKeyName.substring(1));
+    }
+    return isValidKeyPath(normalizedKeyName.substring(1));
+  }
+
+  /**
+   * Whether the pathname is valid.  Check key names which contain a
+   * ":", ".", "..", "//", "". If it has any of these characters throws
+   * OMException, else return the path.
+   */
+  private static String isValidKeyPath(String path) throws OMException {
+    boolean isValid = true;
+
+    // If keyName is empty string throw error.
+    if (path.length() == 0) {
+      throw new OMException("Invalid KeyPath, empty keyName" + path,
+          INVALID_KEY_NAME);
+    } else if(path.startsWith("/")) {
+      isValid = false;
+    } else {
+      // Check for ".." "." ":" "/"
+      String[] components = StringUtils.split(path, '/');
+      for (int i = 0; i < components.length; i++) {
+        String element = components[i];
+        if (element.equals(".") ||
+            (element.contains(":")) ||
+            (element.contains("/") || element.equals(".."))) {
+          isValid = false;
+          break;
+        }
+
+        // The string may end with a /, but not have

Review comment:
       Does this mean we are allowing trailing `/`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 merged pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 merged pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456631086



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -327,7 +344,164 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+
+    keyName = "dir1/dir2/dir3/file1";
+    createAndCheck(keyName);
+
+    // Key with leading '/'.
+    String keyName = "/a/b/c/file1";
+    createAndCheck(keyName);
+
+    // Commit openKey entry.
+    TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+        keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+    // Now create another file in same dir path.
+    keyName = "/a/b/c/file2";
+    createAndCheck(keyName);
+
+    // Create key with multiple /'s
+    // converted to a/b/c/file5
+    keyName = "///a/b///c///file5";
+    createAndCheck(keyName);
+
+    // converted to a/b/c/.../file3
+    keyName = "///a/b///c//.../file3";
+    createAndCheck(keyName);
+
+    // converted to r1/r2
+    keyName = "././r1/r2/";
+    createAndCheck(keyName);
+
+    // converted to ..d1/d2/d3
+    keyName = "..d1/d2/d3/";
+    createAndCheck(keyName);
+
+    // Create a file, where a file already exists in the path.
+    // Now try with a file exists in path. Should fail.
+    keyName = "/a/b/c/file1/file3";
+    checkNotAFile(keyName);
+
+    // Empty keyName.
+    keyName = "";
+    checkNotAValidPath(keyName);
+
+    // Key name ends with /
+    keyName = "/a/./";
+    checkNotAValidPath(keyName);
+
+    keyName = "/////";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b/c";

Review comment:
       Also let's add a check for `../a/b`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r454102604



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -328,6 +342,73 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithIntermediateDir() throws Exception {
+
+    String keyName = "a/b/c/file1";
+    OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+    omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+    OMClientResponse omClientResponse =
+        omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+        100L, ozoneManagerDoubleBufferHelper);
+
+    Assert.assertEquals(omClientResponse.getOMResponse().getStatus(), OK);
+
+    Path keyPath = Paths.get(keyName);
+
+    // Check intermediate paths are created
+    keyPath = keyPath.getParent();
+    while(keyPath != null) {
+      Assert.assertNotNull(omMetadataManager.getKeyTable().get(

Review comment:
       Added test




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-659696446


   Thank You @arp7 and @avijayanhwx for the review.
   I have addressed review comments.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456675451



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
##########
@@ -91,6 +99,19 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     if(checkKeyNameEnabled){
       OmUtils.validateKeyName(keyArgs.getKeyName());
     }
+
+    String keyPath = keyArgs.getKeyName();
+    if (ozoneManager.getEnableFileSystemPaths()) {
+      // If enabled, disallow keys with trailing /. As in fs semantics
+      // directories end with trailing /.
+      keyPath = validateAndNormalizeKey(
+          ozoneManager.getEnableFileSystemPaths(), keyPath);
+      if (keyPath.endsWith("/")) {
+        throw new OMException("Invalid KeyPath: " + keyPath,
+            OMException.ResultCodes.INVALID_KEY_NAME);

Review comment:
       Done

##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##########
@@ -141,7 +141,6 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
               ozoneManager.getPreallocateBlocksMax(),
               ozoneManager.isGrpcBlockTokenEnabled(),
               ozoneManager.getOMNodeId());
-

Review comment:
       Removed it




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456633686



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.ozone.om.request;
+
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+import static org.apache.hadoop.ozone.om.request.OMClientRequest.validateAndNormalizeKey;
+import static org.junit.Assert.fail;
+
+/**
+ * Class to test normalize paths.
+ */
+public class TestNormalizePaths {
+
+  @Rule
+  public ExpectedException exceptionRule = ExpectedException.none();
+
+  @Test
+  public void testNormalizePathsEnabled() throws Exception {
+
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "/a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "////a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "////a/b/////c/d"));
+    Assert.assertEquals("a/b/c/...../d",
+        validateAndNormalizeKey(true, "////a/b/////c/...../d"));

Review comment:
       Bharat, more than two dots should normalize to itself. In Unix I can create a dir named `...` or `.....`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456809378



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.ozone.om.request;
+
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+import static org.apache.hadoop.ozone.om.request.OMClientRequest.validateAndNormalizeKey;
+import static org.junit.Assert.fail;
+
+/**
+ * Class to test normalize paths.
+ */
+public class TestNormalizePaths {
+
+  @Rule
+  public ExpectedException exceptionRule = ExpectedException.none();
+
+  @Test
+  public void testNormalizePathsEnabled() throws Exception {
+
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "/a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "////a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "////a/b/////c/d"));
+    Assert.assertEquals("a/b/c/...../d",
+        validateAndNormalizeKey(true, "////a/b/////c/...../d"));

Review comment:
       Right, I got confused.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r454102550



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
##########
@@ -221,8 +233,45 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
       OmBucketInfo bucketInfo = omMetadataManager.getBucketTable().get(
           omMetadataManager.getBucketKey(volumeName, bucketName));
 
+      boolean createIntermediateDir =
+          ozoneManager.getConfiguration().getBoolean(
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY,
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY_DEFAULT);
+
+      // If FILE_EXISTS we just override like how we used to do for Key Create.
+      List< OzoneAcl > inheritAcls;
+      if (createIntermediateDir) {
+        OMFileRequest.OMPathInfo pathInfo =
+            OMFileRequest.verifyFilesInPath(omMetadataManager, volumeName,
+                bucketName, keyName, Paths.get(keyName));
+        OMFileRequest.OMDirectoryResult omDirectoryResult =
+            pathInfo.getDirectoryResult();
+        inheritAcls = pathInfo.getAcls();
+
+        // Check if a file or directory exists with same key name.
+        if (omDirectoryResult == DIRECTORY_EXISTS) {
+          throw new OMException("Can not write to directory: " + keyName,

Review comment:
       Done

##########
File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
##########
@@ -239,4 +239,11 @@ private OMConfigKeys() {
           "ozone.om.keyname.character.check.enabled";
   public static final boolean OZONE_OM_KEYNAME_CHARACTER_CHECK_ENABLED_DEFAULT =
           false;
+
+  // This config needs to be enabled, when S3G created objects will be used
+  // FileSystem.

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-659696446


   Thank You @arp7 and @avijayanhwx for the review.
   I have addressed review comments.
   
   This PR only handles KeyCreate when ozone.om.enable.filesystem.paths is enabled. (Not changed behavior of actual KeyCreate)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455955940



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -265,4 +272,30 @@ public AuditMessage buildAuditMessage(AuditAction op,
     auditMap.put(OzoneConsts.VOLUME, volume);
     return auditMap;
   }
+
+
+  public static String getNormalizedKey(boolean enableFileSystemPaths,
+      String keyName) {
+    if (enableFileSystemPaths) {
+      return getNormalizedKey(keyName);
+    } else {
+      return keyName;
+    }
+  }
+
+  @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+  public static String getNormalizedKey(String keyName) {
+    String normalizedKeyName;
+    if (keyName.startsWith(OM_KEY_PREFIX)) {
+      normalizedKeyName = Paths.get(keyName).toUri().normalize().getPath();
+    } else {
+      normalizedKeyName = Paths.get(OM_KEY_PREFIX, keyName).toUri()
+          .normalize().getPath();
+    }
+    if (!keyName.equals(normalizedKeyName)) {
+      LOG.debug("Normalized key {} to {} ", keyName,

Review comment:
       Thanks for adding this log message. It will be super useful!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-659696446


   Thank You @arp7 and @avijayanhwx for the review.
   I have addressed review comments.
   
   This PR only modifies behavior of KeyCreate when ozone.om.enable.filesystem.paths is enabled. (Not changed the default behavior of actual KeyCreate)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
arp7 commented on pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-657661338


   The approach looks really good. It is surprisingly concise and elegant because I was expecting it to be a lot more code! Added a few review comments.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-660355893


   > Few minor comments, and couple of questions on the expected normalizations.
   > 
   > Overall the change looks good. Thanks for adding comprehensive test cases.
   
   Thank You @arp7 for the review. addressed review comments.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455231005



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -265,4 +272,24 @@ public AuditMessage buildAuditMessage(AuditAction op,
     auditMap.put(OzoneConsts.VOLUME, volume);
     return auditMap;
   }
+
+  @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+  public static String getNormalizedKey(boolean enableFileSystemPaths,

Review comment:
       What about moving the 'enableFileSystemPaths' flag outside, and call the method conditionally? In that way, the getNormalizedKey will stay a simple util function without knowing about application logic. 

##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -265,4 +272,24 @@ public AuditMessage buildAuditMessage(AuditAction op,
     auditMap.put(OzoneConsts.VOLUME, volume);
     return auditMap;
   }
+
+  @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+  public static String getNormalizedKey(boolean enableFileSystemPaths,
+      String keyName) {
+    if (enableFileSystemPaths) {
+      String normalizedKeyName;
+      if (keyName.startsWith(OM_KEY_PREFIX)) {
+        normalizedKeyName = Paths.get(keyName).toUri().normalize().getPath();
+      } else {
+        normalizedKeyName = Paths.get(OM_KEY_PREFIX, keyName).toUri()
+            .normalize().getPath();
+      }
+      if (!keyName.equals(normalizedKeyName)) {
+        LOG.debug("Normalized key {} to {} ", keyName, normalizedKeyName);

Review comment:
       Can we print normalizedKeyName.substring(1) given we return that?

##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.ozone.om.request;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.ozone.om.request.OMClientRequest.getNormalizedKey;
+
+/**
+ * Class to test normalize paths.
+ */
+public class TestNormalizePaths {
+
+  @Test
+  public void testNormalizePathsEnabled() {
+
+    Assert.assertEquals("a/b/c/d",
+        getNormalizedKey(true, "a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        getNormalizedKey(true, "/a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        getNormalizedKey(true, "////a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        getNormalizedKey(true, "////a/b/////c/d"));
+    Assert.assertEquals("a/b/c/...../d",
+        getNormalizedKey(true, "////a/b/////c/...../d"));
+    Assert.assertEquals("a/b/d",
+        getNormalizedKey(true, "/a/b/c/../d"));
+    Assert.assertEquals("a/d",

Review comment:
       Nit. Can we add a test for key with length = 1? Since we return .substring(1) in the method, trying to be cautious here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455236059



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -265,4 +272,24 @@ public AuditMessage buildAuditMessage(AuditAction op,
     auditMap.put(OzoneConsts.VOLUME, volume);
     return auditMap;
   }
+
+  @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+  public static String getNormalizedKey(boolean enableFileSystemPaths,

Review comment:
       That requires in all over the code, we need to check this flag and call this. I feel this is better way.
   If needed I can move the code of actual Normalize, to another utility if you expect this to be used else where.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455968524



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##########
@@ -79,15 +79,11 @@ public static OMPathInfo verifyFilesInPath(
       String dbDirKeyName = omMetadataManager.getOzoneDirKey(volumeName,
           bucketName, pathName);
 
-      if (omMetadataManager.getKeyTable().isExist(dbKeyName)) {

Review comment:
       This change can be reverted as now we don't allow leading / in path names when OMConfig enable.filesystems is set to true. Will updated in my next PR update.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456668701



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
##########
@@ -327,7 +344,164 @@ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
         .setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
         .setClientId(UUID.randomUUID().toString())
         .setCreateKeyRequest(createKeyRequest).build();
+  }
+
+  @Test
+  public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+    OzoneConfiguration configuration = new OzoneConfiguration();
+    configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+    when(ozoneManager.getConfiguration()).thenReturn(configuration);
+    when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+    // Add volume and bucket entries to DB.
+    addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager);
+
+
+    keyName = "dir1/dir2/dir3/file1";
+    createAndCheck(keyName);
+
+    // Key with leading '/'.
+    String keyName = "/a/b/c/file1";
+    createAndCheck(keyName);
+
+    // Commit openKey entry.
+    TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+        keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+    // Now create another file in same dir path.
+    keyName = "/a/b/c/file2";
+    createAndCheck(keyName);
+
+    // Create key with multiple /'s
+    // converted to a/b/c/file5
+    keyName = "///a/b///c///file5";
+    createAndCheck(keyName);
+
+    // converted to a/b/c/.../file3
+    keyName = "///a/b///c//.../file3";
+    createAndCheck(keyName);
+
+    // converted to r1/r2
+    keyName = "././r1/r2/";
+    createAndCheck(keyName);
+
+    // converted to ..d1/d2/d3
+    keyName = "..d1/d2/d3/";
+    createAndCheck(keyName);
+
+    // Create a file, where a file already exists in the path.
+    // Now try with a file exists in path. Should fail.
+    keyName = "/a/b/c/file1/file3";
+    checkNotAFile(keyName);
+
+    // Empty keyName.
+    keyName = "";
+    checkNotAValidPath(keyName);
+
+    // Key name ends with /
+    keyName = "/a/./";
+    checkNotAValidPath(keyName);
+
+    keyName = "/////";
+    checkNotAValidPath(keyName);
+
+    keyName = "../../b/c";

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: Keyput

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r454001878



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
##########
@@ -221,8 +233,45 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
       OmBucketInfo bucketInfo = omMetadataManager.getBucketTable().get(
           omMetadataManager.getBucketKey(volumeName, bucketName));
 
+      boolean createIntermediateDir =
+          ozoneManager.getConfiguration().getBoolean(
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY,
+              OZONE_OM_CREATE_INTERMEDIATE_DIRECTORY_DEFAULT);
+
+      // If FILE_EXISTS we just override like how we used to do for Key Create.
+      List< OzoneAcl > inheritAcls;
+      if (createIntermediateDir) {
+        OMFileRequest.OMPathInfo pathInfo =
+            OMFileRequest.verifyFilesInPath(omMetadataManager, volumeName,
+                bucketName, keyName, Paths.get(keyName));
+        OMFileRequest.OMDirectoryResult omDirectoryResult =
+            pathInfo.getDirectoryResult();
+        inheritAcls = pathInfo.getAcls();
+
+        // Check if a file or directory exists with same key name.
+        if (omDirectoryResult == DIRECTORY_EXISTS) {
+          throw new OMException("Can not write to directory: " + keyName,
+              NOT_A_FILE);
+        } else
+          if (omDirectoryResult == FILE_EXISTS_IN_GIVENPATH) {
+            throw new OMException("Can not create file: " + keyName +
+                " as there is already file in the given path", NOT_A_FILE);
+          }
+
+        missingParentInfos = OMDirectoryCreateRequest
+            .getAllParentInfo(ozoneManager, keyArgs,
+                pathInfo.getMissingParents(), inheritAcls, trxnLogIndex);
+
+        // Add cache entries for the prefix directories.

Review comment:
       Yes, there will be some intermediate directories left in DB, key commit failed. There will be intermediate directories created.
   But I don't see that causing an issue. But cleaner thing is cleaning up entries. 
   But this is a common issue for FileCreate and KeyCreate.
   
   And in the case lets say key create itself failed in DB flush, we terminate DB.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455445455



##########
File path: hadoop-hdds/common/src/main/resources/ozone-default.xml
##########
@@ -2478,4 +2478,18 @@
       rules in Amazon S3's object key naming guide.
     </description>
   </property>
+
+  <property>
+    <name>ozone.om.enable.filesystem.paths</name>
+    <tag>OZONE, OM</tag>
+    <value>false</value>
+    <description>If true, OM will normalize the key paths and also create

Review comment:
       I would rephrase first sentence a bit to state that key names will be interpreted as file system paths. `/` will be treated as a special character and paths will be normalized and must follow Unix filesystem path naming conventions. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r456666267



##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.ozone.om.request;
+
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+import static org.apache.hadoop.ozone.om.request.OMClientRequest.validateAndNormalizeKey;
+import static org.junit.Assert.fail;
+
+/**
+ * Class to test normalize paths.
+ */
+public class TestNormalizePaths {
+
+  @Rule
+  public ExpectedException exceptionRule = ExpectedException.none();
+
+  @Test
+  public void testNormalizePathsEnabled() throws Exception {
+
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "/a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "////a/b/c/d"));
+    Assert.assertEquals("a/b/c/d",
+        validateAndNormalizeKey(true, "////a/b/////c/d"));
+    Assert.assertEquals("a/b/c/...../d",
+        validateAndNormalizeKey(true, "////a/b/////c/...../d"));

Review comment:
       yes, you are right. It is the samething right, we are checking a/b/c/...../d, just it has removed additional and leading /




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org