[jira] [Commented] (HADOOP-13448) S3Guard: Define MetadataStore interface.

["Aaron Fabbri (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415658#comment-15415658 ] 

Aaron Fabbri commented on HADOOP-13448:
---------------------------------------

To start the interface discussion:

I was looking at the two ConsistentStore/MetadataStore interfaces we posted to HADOOP-13345, and had a question:  Why does your {{DynamoDBConsistentStore#save()}} implementation walk the path to the root and save all ancestor paths as well?  I understand that this is mirroring what the S3A client does.  I guess I preferred leaving the enumeration of ancestor directories that need to be created to the client.  The case I'm thinking of is a workload that does this:

{code}
path = "/dirs"
for x in range(0,10) :
  path += "/dir" + str(x)
  s3aFS.mkdirs(path)
{code}

In this case, we're making N directories that get deeper and deeper with the same ancestors.  This seems like an theta(N^2) cost with your prototype, but only N cost with ours.  The downside to our approach is that the client code has to explicitly track the missing ancestors that need to be passed to the MetadataStore.  In our patch it looked like this at the end of {{innerMkdirs()}}:

{code}
        } catch (FileNotFoundException fnfe) {
          instrumentation.errorIgnored();
          if (dirsToCreate != null)
            dirsToCreate.add(fPart);
        }
        fPart = fPart.getParent();
      } while (fPart != null);

      String key = pathToKey(f);
      createFakeDirectory(key);
      metadataStoreMkdirs(dirsToCreate, permission);
      return true;
    }
  }
{code}

I might be missing something here, let me know what you think.

> S3Guard: Define MetadataStore interface.
> ----------------------------------------
>
>                 Key: HADOOP-13448
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13448
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>
> Define the common interface for metadata store operations.  This is the interface that any metadata back-end must implement in order to integrate with S3Guard.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org