You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Aaron Fabbri (JIRA)" <ji...@apache.org> on 2017/04/04 01:12:41 UTC
[jira] [Comment Edited] (HADOOP-14226) S3Guard:
DynamoDBMetadataStore::move() should populate ancestor directories of
destination paths
[ https://issues.apache.org/jira/browse/HADOOP-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954432#comment-15954432 ]
Aaron Fabbri edited comment on HADOOP-14226 at 4/4/17 1:12 AM:
---------------------------------------------------------------
This patch looks good [~liuml07]. I changed the test code some as I was working through the code. I wanted a test case with a bit more directory depth, and fails before, but succeeds after, the change to {{DynamoDBMetadataStore}}. Attaching v2 patch which just reworks the test case some. I had to avoid doing a put() of the dest path before the move, since put() would (correctly) create the ancestor directory entries. This test depends on move() to do it, so it fails without your change and succeeds with it.
If you are ok with my test changes I can commit the patch after a complete test run.
was (Author: fabbri):
This patch looks good [~liuml07]. I wanted a test case that failed before, but succeeded after, the change to {{DynamoDBMetadataStore}}. Attaching v2 patch which just reworks the test case some. I had to avoid doing a put() of the dest path before the move, since put() would (correctly) create the ancestor directory entries. This test depends on move() to do it, so it fails without your change and succeeds with it.
If you are ok with my test changes I can commit the patch after a complete test run.
> S3Guard: DynamoDBMetadataStore::move() should populate ancestor directories of destination paths
> ------------------------------------------------------------------------------------------------
>
> Key: HADOOP-14226
> URL: https://issues.apache.org/jira/browse/HADOOP-14226
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Mingliang Liu
> Assignee: Mingliang Liu
> Attachments: HADOOP-14226-HADOOP-13345.000.patch, HADOOP-14226-HADOOP-13345.001.patch, HADOOP-14226-HADOOP-13345.002.patch
>
>
> UPDATE: Instead of changing the test, in this JIRA we make sure for each path to put, DDBMS::move() has records of each directory up to the root. See [~fabbri]'s comment in this JIRA for more detail.
> After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not cleaned up. There is a call to {{clearMetadataStore(ms, count);}} in the finally clause though. The reason is that, the internally called method {{DynamoDBMetadataStore::deleteSubtree()}} is assuming there should be an item for the parent dest path:
> {code}
> parent=/fake-bucket, child=moved-here, is_dir=true
> {code}
> In DynamoDBMetadataStore implementation, we assume that _if a path exists, all its ancestors will also exist in the table_. We need to pre-create dest path to maintain this invariant so that test data can be cleaned up successfully.
> I think there may be other tests with the same problem. Let's identify/address them separately.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org