[jira] [Commented] (HADOOP-16423) S3Guarld fsck: Check metadata consistency between S3 and metadatastore (log)

["Gabor Bota (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16895409#comment-16895409 ] 

Gabor Bota commented on HADOOP-16423:
-------------------------------------

To implement:
* only check flag will work, no fix flag yet in CLI
* should be able to run from a defined path, not just from the root
* VersionID is tricky. We need to do a HEAD request as we can not get the versionID with the listing of the directory. This should be *off* by default, and maybe we need a flag to turn it on. But the user should know that this will take time to check as we only check listing when walking the tree.


Tests:
* Check flag must be accepted in S3GuardTool.Fsck (If there's no flag, the check should run by default)
* Path string can be passed from the CLI. (If there's no path string, the root is defined by default)
* Unit test for S3GuardFsck: being able to find all violations. (13 checks, more info in the PR's {{s3guard.S3GuardFsck.Violation}} enum.)
* CLI integration test: create a bunch of files with violations (inconsistencies) and check if we have the right worlds (enum names, so no string literals in tests!) output string.

> S3Guarld fsck: Check metadata consistency between S3 and metadatastore (log)
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-16423
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16423
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.0
>            Reporter: Gabor Bota
>            Assignee: Gabor Bota
>            Priority: Major
>
> This part is only for logging the inconsistencies.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org