You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Aaron Fabbri (JIRA)" <ji...@apache.org> on 2017/10/11 21:52:00 UTC
[jira] [Comment Edited] (HADOOP-13974) S3a CLI to support
list/purge of pending multipart commits
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201032#comment-16201032 ]
Aaron Fabbri edited comment on HADOOP-13974 at 10/11/17 9:51 PM:
-----------------------------------------------------------------
Attaching v1 patch.
Some caveats I'd like feedback on:
- The `hadoop s3guard uploads -abort` command is dangerous, especially if there are MPU commits in flight. I feel like this could use a failsafe "are you sure" prompt. I'm leaning towards implementing that when the specified age is less than one day. Presumably this is "safer" if your filter only matches older upload parts. Or, we could add a "-force" option to override it.
- Also think a disclaimer about clock synchronization might be warranted in the docs. S3's MPU list gives an "initiated time" which I use here. I'm wondering if this is server side or client side?
- There is a test gap.. I tested the age-based filtering by hand but don't have an automated test for it. v2 patch should probably have at least a very basic test of the age filtering.
was (Author: fabbri):
Attaching v1 patch.
Some caveats I'd like feedback on:
- The `hadoop s3guard uploads -abort` command is dangerous, especially if there are MPU commits in flight. I feel like this could use a failsafe "are you sure" prompt. I'm leaning towards implementing that when the specified age is less than one day. Presumably this is "safer" if your filter only matches older upload parts.
- Also think a disclaimer about clock synchronization might be warranted in the docs. S3's MPU list gives an "initiated time" which I use here. I'm wondering if this is server side or client side?
- There is a test gap.. I tested the age-based filtering by hand but don't have an automated test for it. v2 patch should probably have at least a very basic test of the age filtering.
> S3a CLI to support list/purge of pending multipart commits
> ----------------------------------------------------------
>
> Key: HADOOP-13974
> URL: https://issues.apache.org/jira/browse/HADOOP-13974
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-beta1
> Reporter: Steve Loughran
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13974.001.patch
>
>
> The S3A CLI will need to be able to list and delete pending multipart commits.
> We can do the cleanup already via fs.s3a properties. The CLI will let scripts stat for outstanding data (have a different exit code) and permit batch jobs to explicitly trigger cleanups.
> This will become critical with the multipart committer, as there's a significantly higher likelihood of commits remaining outstanding.
> We may also want to be able to enumerate/cancel all pending commits in the FS tree
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org