You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2017/02/02 22:04:51 UTC
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for
zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850578#comment-15850578 ]
Hadoop QA commented on HADOOP-13786:
------------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 11 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 11s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 55s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 34s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 6s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 7s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 6s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 54s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 18s{color} | {color:red} root generated 3 new + 700 unchanged - 0 fixed = 703 total (was 700) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 37s{color} | {color:orange} root: The patch generated 80 new + 83 unchanged - 4 fixed = 163 total (was 87) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 30s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 52 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 54s{color} | {color:red} hadoop-tools/hadoop-aws generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 21s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 12s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 36s{color} | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 22s{color} | {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
| | org.apache.hadoop.fs.s3a.S3AFileSystem$WriteOperationHelper defines equals and uses Object.hashCode() At S3AFileSystem.java:Object.hashCode() At S3AFileSystem.java:[line 2483] |
| | Using pointer equality to compare a java.util.ArrayList<org.apache.hadoop.fs.s3a.commit.FileCommitActions$CommitFileOutcome> with a FileCommitActions$CommitFileOutcome in org.apache.hadoop.fs.s3a.commit.FileCommitActions$CommitAllFilesOutcome.add(FileCommitActions$CommitFileOutcome) At FileCommitActions.java:a java.util.ArrayList<org.apache.hadoop.fs.s3a.commit.FileCommitActions$CommitFileOutcome> with a FileCommitActions$CommitFileOutcome in org.apache.hadoop.fs.s3a.commit.FileCommitActions$CommitAllFilesOutcome.add(FileCommitActions$CommitFileOutcome) At FileCommitActions.java:[line 274] |
| | Should org.apache.hadoop.fs.s3a.commit.S3AOutputCommitter$DurationInfo be a _static_ inner class? At S3AOutputCommitter.java:inner class? At S3AOutputCommitter.java:[lines 349-366] |
| Failed junit tests | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13786 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12850677/HADOOP-13786-HADOOP-13345-004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
| uname | Linux 4b9ec73a06cc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | HADOOP-13345 / a4ae955 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/11564/artifact/patchprocess/diff-compile-javac-root.txt |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/11564/artifact/patchprocess/diff-checkstyle-root.txt |
| mvnsite | https://builds.apache.org/job/PreCommit-HADOOP-Build/11564/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-aws.txt |
| whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/11564/artifact/patchprocess/whitespace-eol.txt |
| findbugs | https://builds.apache.org/job/PreCommit-HADOOP-Build/11564/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html |
| javadoc | https://builds.apache.org/job/PreCommit-HADOOP-Build/11564/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/11564/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/11564/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/11564/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
> ------------------------------------------------------------------------
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, HADOOP-13786-HADOOP-13345-004.patch
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the presence of failures". Implement it, including whatever is needed to demonstrate the correctness of the algorithm. (that is, assuming that s3guard provides a consistent view of the presence/absence of blobs, show that we can commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie. not visible until the close()), if we need to use that to allow us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org