You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/08/26 16:11:25 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #2247: HADOOP-17199. S3A Directory Marker HADOOP-13230 backport
steveloughran opened a new pull request #2247:
URL: https://github.com/apache/hadoop/pull/2247
This backports the listing-side changes of HADOOP-13230.
With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.
It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"
The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.
Read the documentation!
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2247: HADOOP-17199. S3A Directory Marker HADOOP-13230 backport to branch-3.1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2247:
URL: https://github.com/apache/hadoop/pull/2247#issuecomment-682673449
failures in hadoop-common, which would seem utterly unrelated
```
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2247/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:[481,16] error: cannot find symbol
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2247/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java:[40,20] error: cannot find symbol
[INFO] 2 errors
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2247: HADOOP-17199. S3A Directory Marker HADOOP-13230 backport to branch-3.1
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #2247:
URL: https://github.com/apache/hadoop/pull/2247
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2247: HADOOP-17199. S3A Directory Marker HADOOP-13230 backport to branch-3.1
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2247:
URL: https://github.com/apache/hadoop/pull/2247#issuecomment-681012368
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 12m 14s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ branch-3.1 Compile Tests _ |
| +0 :ok: | mvndep | 2m 59s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 2m 21s | root in branch-3.1 failed. |
| -1 :x: | compile | 1m 7s | root in branch-3.1 failed. |
| -0 :warning: | checkstyle | 0m 50s | The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 39s | hadoop-common in branch-3.1 failed. |
| -1 :x: | mvnsite | 0m 33s | hadoop-aws in branch-3.1 failed. |
| -1 :x: | shadedclient | 2m 57s | branch has errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 24s | hadoop-common in branch-3.1 failed. |
| -1 :x: | javadoc | 0m 11s | hadoop-aws in branch-3.1 failed. |
| +0 :ok: | spotbugs | 4m 25s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 39s | hadoop-common in branch-3.1 failed. |
| -1 :x: | findbugs | 0m 12s | hadoop-aws in branch-3.1 failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 40s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 36s | hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 0m 10s | hadoop-aws in the patch failed. |
| -1 :x: | compile | 1m 3s | root in the patch failed. |
| -1 :x: | javac | 1m 3s | root in the patch failed. |
| -0 :warning: | checkstyle | 0m 43s | The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 38s | hadoop-common in the patch failed. |
| -1 :x: | mvnsite | 0m 12s | hadoop-aws in the patch failed. |
| -1 :x: | whitespace | 0m 0s | The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | shadedclient | 0m 49s | patch has errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 15s | hadoop-common in the patch failed. |
| -1 :x: | javadoc | 0m 12s | hadoop-aws in the patch failed. |
| -1 :x: | findbugs | 0m 37s | hadoop-common in the patch failed. |
| -1 :x: | findbugs | 0m 11s | hadoop-aws in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 36s | hadoop-common in the patch failed. |
| -1 :x: | unit | 0m 11s | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 33m 7s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2247 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux ea6d61c7ffc5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.1 / 0808cb6 |
| Default Java | Oracle Corporation-9-internal+0-2016-04-14-195246.buildd.src |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/branch-compile-root.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/buildtool-branch-checkstyle-root.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-compile-root.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-compile-root.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/buildtool-patch-checkstyle-root.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-findbugs-hadoop-common-project_hadoop-common.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/testReport/ |
| Max. process+thread count | 151 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2247/1/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2247: HADOOP-17199. S3A Directory Marker HADOOP-13230 backport to branch-3.1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2247:
URL: https://github.com/apache/hadoop/pull/2247#issuecomment-683276805
did the local build of a full distro; ran aws tests, all good. so merging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran edited a comment on pull request #2247: HADOOP-17199. S3A Directory Marker HADOOP-13230 backport to branch-3.1
Posted by GitBox <gi...@apache.org>.
steveloughran edited a comment on pull request #2247:
URL: https://github.com/apache/hadoop/pull/2247#issuecomment-680988478
Test run against London with s3guard. Failures as the SDK/tests aren't ready for 0 provisioned IO Capacity; and something is up with AssumedRole permissions. There's changes in that test for branch-3.2, which I assume "make it go away". This patch doesn't go near s3guard
```
[ERROR] Failures:
[ERROR] ITestDynamoDBMetadataStore.testProvisionTable:588->Assert.assertEquals:542->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 expected:<640> but was:<320>
[ERROR] Errors:
[ERROR] ITestAssumeRole.testAssumeRolePoliciesOverrideRolePerms:336 » AWSServiceIO ini...
[ERROR] ITestAssumeRole.testAssumeRoleRestrictedPolicyFS:309 » AWSServiceIO initTable ...
[ERROR] ITestAssumeRole.testAssumedInvalidRole:150->lambda$testAssumedInvalidRole$1:152 » IllegalArgument
[ERROR] ITestAssumeRole.testPartialDelete:727->executePartialDelete:759 » AWSServiceIO
[ERROR] ITestAssumeRole.testPartialDeleteSingleDelete:735->executePartialDelete:759 » AWSServiceIO
[ERROR] ITestAssumeRole.testReadOnlyOperations:355 » AWSServiceIO initTable on stevel-...
[ERROR] ITestAssumeRole.testRestrictedCommitActions:621 » AWSServiceIO initTable on st...
[ERROR] ITestAssumeRole.testRestrictedRename:435->executeRestrictedRename:469 » AWSServiceIO
[ERROR] ITestAssumeRole.testRestrictedSingleDeleteRename:444->executeRestrictedRename:469 » AWSServiceIO
[ERROR] ITestAssumeRole.testRestrictedWriteSubdir:405 » AWSServiceIO initTable on stev...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:231->AbstractS3GuardToolTestBase.run:95 » IllegalArgument
[ERROR] ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle:224->AbstractS3GuardToolTestBase.exec:286->AbstractS3GuardToolTestBase.exec:308 » AWSServiceIO
[ERROR] ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:231->AbstractS3GuardToolTestBase.run:95 » IllegalArgument
```
In its favour -full scale test with 6 threads *only took 15 minutes*. How I miss that on trunk. We have added too much code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2247: HADOOP-17199. S3A Directory Marker HADOOP-13230 backport to branch-3.1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2247:
URL: https://github.com/apache/hadoop/pull/2247#issuecomment-680988478
Test run against London with s3guard. Failures as the SDK/tests aren't ready for 0 provisioned IO Capacity; and something is up with AssumedRole permissions. There's changes in that test for branch-3.2, which I assume "make it go away". This patch doesn't go near s3guard
```
[ERROR] Failures:
[ERROR] ITestDynamoDBMetadataStore.testProvisionTable:588->Assert.assertEquals:542->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 expected:<640> but was:<320>
[ERROR] Errors:
[ERROR] ITestAssumeRole.testAssumeRolePoliciesOverrideRolePerms:336 » AWSServiceIO ini...
[ERROR] ITestAssumeRole.testAssumeRoleRestrictedPolicyFS:309 » AWSServiceIO initTable ...
[ERROR] ITestAssumeRole.testAssumedInvalidRole:150->lambda$testAssumedInvalidRole$1:152 » IllegalArgument
[ERROR] ITestAssumeRole.testPartialDelete:727->executePartialDelete:759 » AWSServiceIO
[ERROR] ITestAssumeRole.testPartialDeleteSingleDelete:735->executePartialDelete:759 » AWSServiceIO
[ERROR] ITestAssumeRole.testReadOnlyOperations:355 » AWSServiceIO initTable on stevel-...
[ERROR] ITestAssumeRole.testRestrictedCommitActions:621 » AWSServiceIO initTable on st...
[ERROR] ITestAssumeRole.testRestrictedRename:435->executeRestrictedRename:469 » AWSServiceIO
[ERROR] ITestAssumeRole.testRestrictedSingleDeleteRename:444->executeRestrictedRename:469 » AWSServiceIO
[ERROR] ITestAssumeRole.testRestrictedWriteSubdir:405 » AWSServiceIO initTable on stev...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestAssumedRoleCommitOperations.setup:81 » AWSServiceIO initTable on stevel-l...
[ERROR] ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:231->AbstractS3GuardToolTestBase.run:95 » IllegalArgument
[ERROR] ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle:224->AbstractS3GuardToolTestBase.exec:286->AbstractS3GuardToolTestBase.exec:308 » AWSServiceIO
[ERROR] ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:231->AbstractS3GuardToolTestBase.run:95 » IllegalArgument
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org