You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/02/09 09:14:04 UTC
[GitHub] [hadoop] mukund-thakur opened a new pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
mukund-thakur opened a new pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838
Testing Bucket used : https://mthakur-data.s3.ap-south-1.amazonaws.com/file2
Ran all the UT's and IT's using default settings with S3guard.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-588444854
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 46s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 29s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 43s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 1m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 2s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 40s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | the patch passed |
| +1 :green_heart: | findbugs | 1m 25s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 35s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 66m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint xml |
| uname | Linux 9de8b593adf5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ec75071 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/8/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/8/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/8/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-585343180
testing myself, setting validation to 0 for the entire test run to field test it better.
one failure so far in the new test
```
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardTtl
[ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.739 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ABucketExistence
[ERROR] testNoBucketProbing(org.apache.hadoop.fs.s3a.ITestS3ABucketExistence) Time elapsed: 1.258 s <<< ERROR!
java.lang.IllegalArgumentException: Path s3a://random-bucket-442f6634-4892-4239-bd8c-ac5a2b0a3700 is not absolute
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:216)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.checkPath(DynamoDBMetadataStore.java:1851)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:718)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:205)
at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getWithTtl(S3Guard.java:900)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2729)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2696)
at org.apache.hadoop.fs.s3a.ITestS3ABucketExistence.lambda$testNoBucketProbing$0(ITestS3ABucketExistence.java:66)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:453)
at org.apache.hadoop.fs.s3a.ITestS3ABucketExistence.testNoBucketProbing(ITestS3ABucketExistence.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
``
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584694342
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 40s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 13s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 17s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 0s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 25s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 62m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux cf0667027565 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cc8ae59 |
| Default Java | 1.8.0_242 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/4/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584831964
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 14s | trunk passed |
| +1 :green_heart: | compile | 0m 33s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-aws: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) |
| +1 :green_heart: | mvnsite | 0m 29s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 25s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 64m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 2732dfab6c4e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e637797 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/5/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/5/testReport/ |
| Max. process+thread count | 425 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584804743
> if closing the `fs` value triggers failures in superclass cleanup, then you are sharing an FS instance between test cases. (i.e you are actually picking up the last one created).
That is fixed now. That was a mistake from my side. Closing "fs" is not causing any problem in superclass cleanup now.
One other thing to notice here is there is only one test case where the 'fs' is actually created. All others are just failure scenarios.
> If you disable caching you should get a new one, which you can then close safely
Already disabled file system caching.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur edited a comment on issue #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur edited a comment on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-588400930
> Two tests are failing. Will debug. Also will rebase from trunk and fix merge conflicts.
Fixed merge conflicts. Above tests are succeeding now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377593684
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
##########
@@ -608,3 +608,27 @@ with HADOOP-15669.
Other options may be added to `fs.s3a.ssl.channel.mode` in the future as
further SSL optimizations are made.
+
+## Tuning S3AFileSystem Initialization.
+Any client using S3AFileSystem has to initialize it by providing a S3 bucket
+and configuration. The init method checks if the bucket provided is valid
+or not which is a slow operation leading poor performance. We can ignore
+bucket validation by configuring `fs.s3a.bucket.probe` as follows:
+
+```xml
+<property>
+ <name>fs.s3a.bucket.probe</name>
+ <value>0</value>
+ <description>
+ The value can be 0, 1 or 2(default). When set to 0, bucket existence
+ check won't be done during initialization thus making it faster.
+ Though it should be noted that if bucket is not available in S3,
Review comment:
Doc changes already present similar to you said. Do you want me to tweak this here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-583830785
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 58s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 27m 6s | trunk passed |
| +1 :green_heart: | compile | 0m 38s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 33s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 8s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 35s | trunk passed |
| +0 :ok: | spotbugs | 1m 22s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 15s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | the patch passed |
| +1 :green_heart: | compile | 0m 34s | the patch passed |
| +1 :green_heart: | javac | 0m 34s | the patch passed |
| -0 :warning: | checkstyle | 0m 26s | hadoop-tools/hadoop-aws: The patch generated 22 new + 15 unchanged - 0 fixed = 37 total (was 15) |
| +1 :green_heart: | mvnsite | 0m 42s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 17m 52s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | the patch passed |
| -1 :x: | findbugs | 1m 22s | hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 78m 33s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Switch statement found in org.apache.hadoop.fs.s3a.S3AFileSystem.doBucketProbing() where default case is missing At S3AFileSystem.java:where default case is missing At S3AFileSystem.java:[lines 463-470] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux d8622c0ac2b8 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d23317b |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/whitespace-eol.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-585086587
All review comments addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377827793
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
Review comment:
yes. This is all just information for people looking at the code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-588444854
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 46s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 29s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 43s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 1m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 2s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 40s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | the patch passed |
| +1 :green_heart: | findbugs | 1m 25s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 35s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 66m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint xml |
| uname | Linux 9de8b593adf5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ec75071 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/8/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/8/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/8/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377168238
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
Review comment:
add a RetryPolicy by looking @ the methods it calls and see what they do
We mustn't have a retry() calling operations which retry themselves anyway, as it explodes the #of retries which take place. It's ok to use once() round either of them, as the exception translating stuff is a no-op on already translated exceptions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377659612
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
Review comment:
After reading some documentation, I understood what you mean to say here. So I have added @Retries.RetryTranslated annotation there. That means since the underlying methods are retried and translated, the caller shouldn't retry or translate again. Please correct me if I am wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377659612
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
Review comment:
After reading some documentation, I understood what you mean to say here. So I have added @Retries.RetryTranslated annotation there. That means since the underling methods are retried and translated, the called shouldn't retry or translate again. Please correct me if I am wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377796949
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.After;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ throw ex;
+ }
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
+ "No such file or directory: " + path,
+ () -> fs.getFileStatus(path));
+
+ Path src = new Path(fs.getUri() + "/testfile");
+ byte[] data = dataset(1024, 'a', 'z');
+ intercept(FileNotFoundException.class,
+ "The specified bucket does not exist",
+ () -> writeDataset(fs, src, data, data.length, 1024 * 1024, true));
+ }
+
+ @Test
+ public void testBucketProbingV1() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 1);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingV2() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 2);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingParameterValidation() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 3);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ configuration.setInt(S3A_BUCKET_PROBE, -1);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Override
+ protected Configuration getConfiguration() {
+ Configuration configuration = super.getConfiguration();
+ S3ATestUtils.disableFilesystemCaching(configuration);
+ return configuration;
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ IOUtils.cleanupWithLogger(getLogger(), fs);
Review comment:
ok. So what's happening then is that your tests are picking up a shared FS instance, not the one you've just configured with different bucket init settings. your tests aren't doing what you think they are
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377170722
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = this.getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ fail("Exception shouldn't have occurred");
+ }
+ assertNotNull("FileSystem should have been initialized", fs);
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
Review comment:
What's the full stack here? Because I don't want the FNFE from a missing path to be confused with a getFileStatus failure, as that could go on to confuse other things
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377555178
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
Review comment:
actually, given it's non-static it will be unique to each test case. You can just override the `teardown()` method and add an `IOUtilcleanupWithLogger(LOG, fs)` & so close the fs variable robustly if it is set. Do call the superclass after.
FWIW, the S3A base test suite already retrieves an FS instance for each test case, so you can pick that up, it's just a bit fiddlier to configure. Don't' worry about it here, but you will eventually have to learn your way around that test code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377166880
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
+ int bucketProbe = this.getConf()
+ .getInt(S3A_BUCKET_PROBE, S3A_BUCKET_PROBE_DEFAULT);
+ Preconditions.checkArgument(bucketProbe >= 0 && bucketProbe <= 2,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2");
+ switch (bucketProbe) {
+ case 0:
+ break;
+ case 1:
+ verifyBucketExists();
+ break;
+ case 2:
+ verifyBucketExistsV2();
+ break;
+ default:
+ break;
Review comment:
Add a comment here saying this won't be reached because of the checks above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-589673298
+1; merged with a couple of final fixups of the test (add @Deprecated) to the test class to shut up checkstyle
thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584801005
OK, production code all LGTM; just that test tuning
if closing the `fs` value triggers failures in superclass cleanup, then you are sharing an FS instance between test cases. (i.e you are actually picking up the last one created). If you disable caching you should get a new one, which you can then close safely
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377829741
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.After;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ throw ex;
+ }
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
+ "No such file or directory: " + path,
+ () -> fs.getFileStatus(path));
+
+ Path src = new Path(fs.getUri() + "/testfile");
+ byte[] data = dataset(1024, 'a', 'z');
+ intercept(FileNotFoundException.class,
+ "The specified bucket does not exist",
+ () -> writeDataset(fs, src, data, data.length, 1024 * 1024, true));
+ }
+
+ @Test
+ public void testBucketProbingV1() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 1);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingV2() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 2);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingParameterValidation() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 3);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ configuration.setInt(S3A_BUCKET_PROBE, -1);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Override
+ protected Configuration getConfiguration() {
+ Configuration configuration = super.getConfiguration();
+ S3ATestUtils.disableFilesystemCaching(configuration);
+ return configuration;
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ IOUtils.cleanupWithLogger(getLogger(), fs);
Review comment:
My tests are running new FS instance only. I confirmed that using the IDE debugger. I think what is happening is, we are calling fs.close() twice one with the shared instance and other on my private instance which is stopping all the services for a particular Fs leading to mismatch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-585351633
Also once I fix that by adding a trailing /, the getFileStatus("/") fails to raise an FNFE, which is because S3Guard is enabled for all buckets on my test setup, *and s3guard DDB will create a stub FS Status on a root entry*.
```
java.lang.AssertionError: Expected a java.io.FileNotFoundException to be thrown, but got the result: : S3AFileStatus{path=s3a://random-bucket-11df1b68-2535-4a9b-9fd5-c6a4d5a6c192/; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:453)
at org.apache.hadoop.fs.s3a.ITestS3ABucketExistence.testNoBucketProbing(ITestS3ABucketExistence.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.lang.Thread.run(Thread.java:748)
```
We could consider shortcutting some of the getFileStatus queries against / in S3A FS itself -it's always a dir after all
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584032513
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 24m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 46s | trunk passed |
| +1 :green_heart: | compile | 0m 36s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 11s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | trunk passed |
| +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 1s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 29s | the patch passed |
| +1 :green_heart: | javac | 0m 29s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 42s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 1m 14s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 25s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 91m 0s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 482fb6d9e33f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d5467d2 |
| Default Java | 1.8.0_242 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/2/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-583830785
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 58s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 27m 6s | trunk passed |
| +1 :green_heart: | compile | 0m 38s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 33s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 8s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 35s | trunk passed |
| +0 :ok: | spotbugs | 1m 22s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 15s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | the patch passed |
| +1 :green_heart: | compile | 0m 34s | the patch passed |
| +1 :green_heart: | javac | 0m 34s | the patch passed |
| -0 :warning: | checkstyle | 0m 26s | hadoop-tools/hadoop-aws: The patch generated 22 new + 15 unchanged - 0 fixed = 37 total (was 15) |
| +1 :green_heart: | mvnsite | 0m 42s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 17m 52s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | the patch passed |
| -1 :x: | findbugs | 1m 22s | hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 78m 33s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Switch statement found in org.apache.hadoop.fs.s3a.S3AFileSystem.doBucketProbing() where default case is missing At S3AFileSystem.java:where default case is missing At S3AFileSystem.java:[lines 463-470] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux d8622c0ac2b8 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d23317b |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/whitespace-eol.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377833699
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.After;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ throw ex;
+ }
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
+ "No such file or directory: " + path,
+ () -> fs.getFileStatus(path));
+
+ Path src = new Path(fs.getUri() + "/testfile");
+ byte[] data = dataset(1024, 'a', 'z');
+ intercept(FileNotFoundException.class,
+ "The specified bucket does not exist",
+ () -> writeDataset(fs, src, data, data.length, 1024 * 1024, true));
+ }
+
+ @Test
+ public void testBucketProbingV1() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 1);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingV2() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 2);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingParameterValidation() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 3);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ configuration.setInt(S3A_BUCKET_PROBE, -1);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Override
+ protected Configuration getConfiguration() {
+ Configuration configuration = super.getConfiguration();
+ S3ATestUtils.disableFilesystemCaching(configuration);
+ return configuration;
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ IOUtils.cleanupWithLogger(getLogger(), fs);
Review comment:
Found the issue. Rather than overriding the teardown method I implemented it which caused the Junit to call teardown() twice thus causing all the above problems.
Sorry My Bad. :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584831964
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 14s | trunk passed |
| +1 :green_heart: | compile | 0m 33s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-aws: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) |
| +1 :green_heart: | mvnsite | 0m 29s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 25s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 64m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 2732dfab6c4e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e637797 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/5/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/5/testReport/ |
| Max. process+thread count | 425 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377590167
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = this.getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
Review comment:
I was worried about this but somehow new conf settings were getting picked up. Need to figure out how. Anyway I have disabled the FilesystemCaching such that we don't see intermittent failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r378419167
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.After;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ throw ex;
+ }
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
+ "No such file or directory: " + path,
+ () -> fs.getFileStatus(path));
+
+ Path src = new Path(fs.getUri() + "/testfile");
+ byte[] data = dataset(1024, 'a', 'z');
+ intercept(FileNotFoundException.class,
+ "The specified bucket does not exist",
+ () -> writeDataset(fs, src, data, data.length, 1024 * 1024, true));
+ }
+
+ @Test
+ public void testBucketProbingV1() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 1);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingV2() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 2);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingParameterValidation() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 3);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ configuration.setInt(S3A_BUCKET_PROBE, -1);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Override
+ protected Configuration getConfiguration() {
+ Configuration configuration = super.getConfiguration();
+ S3ATestUtils.disableFilesystemCaching(configuration);
+ return configuration;
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ IOUtils.cleanupWithLogger(getLogger(), fs);
Review comment:
no worries
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377593111
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = this.getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ fail("Exception shouldn't have occurred");
+ }
+ assertNotNull("FileSystem should have been initialized", fs);
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
Review comment:
added the text in contained for verification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur edited a comment on issue #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur edited a comment on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584804743
> if closing the `fs` value triggers failures in superclass cleanup, then you are sharing an FS instance between test cases. (i.e you are actually picking up the last one created).
That is fixed now. That was a mistake from my side. Closing "fs" is not causing any problem in superclass cleanup now.
One other thing to notice here is there is only one test case where the 'fs' is actually created. All others are just failure scenarios.
> If you disable caching you should get a new one, which you can then close safely
Already disabled file system caching.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377172646
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = this.getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
Review comment:
You get a problem in this code because FileSystem.cache() will cache on the URI only; if there's an FS in the cache, your settings aren't picked up -you will always get the previous instance. That often causes intermittent problems with test runs.
1. Use S3ATestUtils.disableFilesystemCaching to turn off caching of the filesystems you get via FileSystem.get
2. and close() them at the end of each test case. You can do with with try/finally or a try-with-resources clause
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377594483
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.After;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ throw ex;
+ }
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
+ "No such file or directory: " + path,
+ () -> fs.getFileStatus(path));
+
+ Path src = new Path(fs.getUri() + "/testfile");
+ byte[] data = dataset(1024, 'a', 'z');
+ intercept(FileNotFoundException.class,
+ "The specified bucket does not exist",
+ () -> writeDataset(fs, src, data, data.length, 1024 * 1024, true));
+ }
+
+ @Test
+ public void testBucketProbingV1() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 1);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingV2() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 2);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingParameterValidation() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 3);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ configuration.setInt(S3A_BUCKET_PROBE, -1);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Override
+ protected Configuration getConfiguration() {
+ Configuration configuration = super.getConfiguration();
+ S3ATestUtils.disableFilesystemCaching(configuration);
+ return configuration;
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ IOUtils.cleanupWithLogger(getLogger(), fs);
Review comment:
Adding this extra cleanup is throwing FileSystem is closed! because of this call AbstractFSContractTestBase.deleteTestDirInTeardown() in the super class teardown after each test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377169124
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = this.getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ fail("Exception shouldn't have occurred");
Review comment:
don't do this...just have the exception thrown all the way up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-585103560
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 45s | trunk passed |
| +1 :green_heart: | compile | 0m 35s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 26s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 16s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 23s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 65m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 75831ca7b3e5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 9709afe |
| Default Java | 1.8.0_242 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/6/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584636545
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 54s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 24s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 15s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 5s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 64m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 70be778f4667 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cc8ae59 |
| Default Java | 1.8.0_242 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/3/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-588323865
Two tests are failing. Will debug. Also will rebase from trunk and fix merge conflicts.
> [INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[ERROR] Tests run: 9, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 48.592 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[ERROR] testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 15.053 s <<< FAILURE!
java.lang.AssertionError:
Expected no results from listLocatedStatus(/), but got 1 elements:
S3ALocatedFileStatus{path=s3a://mthakur-data/fork-0002; isDirectory=true; modification_time=1582129763970; access_time=0; owner=mthakur; group=mthakur; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false}[eTag='', versionId='']
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.assertNoElements(AbstractContractRootDirectoryTest.java:218)
at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:202)
at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[ERROR] testSimpleRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 0.819 s <<< FAILURE!
java.lang.AssertionError:
listStatus(/) vs listLocatedStatus(/) with
listStatus =S3AFileStatus{path=s3a://mthakur-data/test; isDirectory=true; modification_time=0; access_time=0; owner=mthakur; group=mthakur; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null listLocatedStatus = S3ALocatedFileStatus{path=s3a://mthakur-data/fork-0002; isDirectory=true; modification_time=1582129765040; access_time=0; owner=mthakur; group=mthakur; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false}[eTag='', versionId='']
S3ALocatedFileStatus{path=s3a://mthakur-data/test; isDirectory=true; modification_time=1582129765040; access_time=0; owner=mthakur; group=mthakur; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false}[eTag='', versionId=''] expected:<1> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testSimpleRootListing(AbstractContractRootDirectoryTest.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377796308
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
##########
@@ -608,3 +608,27 @@ with HADOOP-15669.
Other options may be added to `fs.s3a.ssl.channel.mode` in the future as
further SSL optimizations are made.
+
+## Tuning S3AFileSystem Initialization.
+Any client using S3AFileSystem has to initialize it by providing a S3 bucket
+and configuration. The init method checks if the bucket provided is valid
+or not which is a slow operation leading poor performance. We can ignore
+bucket validation by configuring `fs.s3a.bucket.probe` as follows:
+
+```xml
+<property>
+ <name>fs.s3a.bucket.probe</name>
+ <value>0</value>
+ <description>
+ The value can be 0, 1 or 2(default). When set to 0, bucket existence
+ check won't be done during initialization thus making it faster.
+ Though it should be noted that if bucket is not available in S3,
Review comment:
no, that's fine
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377833699
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.After;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ throw ex;
+ }
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
+ "No such file or directory: " + path,
+ () -> fs.getFileStatus(path));
+
+ Path src = new Path(fs.getUri() + "/testfile");
+ byte[] data = dataset(1024, 'a', 'z');
+ intercept(FileNotFoundException.class,
+ "The specified bucket does not exist",
+ () -> writeDataset(fs, src, data, data.length, 1024 * 1024, true));
+ }
+
+ @Test
+ public void testBucketProbingV1() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 1);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingV2() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 2);
+ intercept(FileNotFoundException.class,
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Test
+ public void testBucketProbingParameterValidation() throws Exception {
+ Configuration configuration = getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 3);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ configuration.setInt(S3A_BUCKET_PROBE, -1);
+ intercept(IllegalArgumentException.class,
+ "Value of " + S3A_BUCKET_PROBE + " should be between 0 to 2",
+ "Should throw IllegalArgumentException",
+ () -> FileSystem.get(uri, configuration));
+ }
+
+ @Override
+ protected Configuration getConfiguration() {
+ Configuration configuration = super.getConfiguration();
+ S3ATestUtils.disableFilesystemCaching(configuration);
+ return configuration;
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ IOUtils.cleanupWithLogger(getLogger(), fs);
Review comment:
Found the issue. Rather than overriding the teardown method I implemented it which caused the Junit to call teardown() twice thus cause all the above problems.
Sorry My Bad. :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377838398
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = this.getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ fail("Exception shouldn't have occurred");
+ }
+ assertNotNull("FileSystem should have been initialized", fs);
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
Review comment:
potentially brittle, but we can deal with that when the text changes. We have found in the past hat any test coded to look for AWS error messages is brittle against SDK.
Lets just go with this and when things break, catch up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-588400930
> Two tests are failing. Will debug. Also will rebase from trunk and fix merge conflicts.
Fixed merge conflicts. Above tests are succeeding now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-589159533
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 42s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 19s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| -1 :x: | javac | 0m 26s | hadoop-tools_hadoop-aws generated 3 new + 15 unchanged - 0 fixed = 18 total (was 15) |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 2 new + 30 unchanged - 0 fixed = 32 total (was 30) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 35s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 12s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 65m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint xml |
| uname | Linux 3f134164c323 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 181e6d0 |
| Default Java | 1.8.0_242 |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/9/artifact/out/diff-compile-javac-hadoop-tools_hadoop-aws.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/9/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/9/testReport/ |
| Max. process+thread count | 425 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/9/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377169421
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = this.getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ fail("Exception shouldn't have occurred");
+ }
+ assertNotNull("FileSystem should have been initialized", fs);
Review comment:
and no need to worry about this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377173220
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
Review comment:
unless you want to share across test cases (you don't) or want to have cleanup in the teardown code, move this into a local variable in each test case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1838: HADOOP-16711 Add
way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584636545
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 54s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 24s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 15s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 5s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 64m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 70be778f4667 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cc8ae59 |
| Default Java | 1.8.0_242 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/3/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377170084
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.UUID;
+
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.apache.hadoop.fs.s3a.Constants.S3A_BUCKET_PROBE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Class to test bucket existence api.
+ * See {@link S3AFileSystem#doBucketProbing()}.
+ */
+public class ITestS3ABucketExistence extends AbstractS3ATestBase {
+
+ private FileSystem fs;
+
+ private final String randomBucket =
+ "random-bucket-" + UUID.randomUUID().toString();
+
+ private final URI uri = URI.create(FS_S3A + "://" + randomBucket);
+
+ @Test
+ public void testNoBucketProbing() throws Exception {
+ Configuration configuration = this.getConfiguration();
+ configuration.setInt(S3A_BUCKET_PROBE, 0);
+ try {
+ fs = FileSystem.get(uri, configuration);
+ } catch (IOException ex) {
+ LOG.error("Exception : ", ex);
+ fail("Exception shouldn't have occurred");
+ }
+ assertNotNull("FileSystem should have been initialized", fs);
+
+ Path path = new Path(uri);
+ intercept(FileNotFoundException.class,
+ () -> fs.getFileStatus(path));
+
+ Path src = new Path(fs.getUri() + "/testfile");
+ byte[] data = dataset(1024, 'a', 'z');
+ intercept(FileNotFoundException.class,
+ () -> writeDataset(fs, src, data, data.length, 1024 * 1024, true));
+ }
+
+ @Test
+ public void testBucketProbingV1() throws Exception {
+ Configuration configuration = this.getConfiguration();
Review comment:
no need for `this.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377517531
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
Review comment:
verifyBucketExists() and verifyBucketExistsV2() are the methods which are getting called from the doBucketProbing() method and I see they are already using invoked which has a retry policy set to TRY_ONCE_THEN_FAIL. Do we need to put explicit retry in this method?
Also both these methods are annotated with RetryTranslated and the documentation of Retried say that if RetryTranslated is used, the called shouldn't perform another layer of retries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur removed a comment on issue #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
mukund-thakur removed a comment on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#issuecomment-584804743
> if closing the `fs` value triggers failures in superclass cleanup, then you are sharing an FS instance between test cases. (i.e you are actually picking up the last one created).
That is fixed now. That was a mistake from my side. Closing "fs" is not causing any problem in superclass cleanup now.
One other thing to notice here is there is only one test case where the 'fs' is actually created. All others are just failure scenarios.
> If you disable caching you should get a new one, which you can then close safely
Already disabled file system caching.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377552989
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
Review comment:
you just need to add the @RetryPolicy on the method based on the inner ones. it's not about actually doing the retries, just declare what it is for people looking at it.
The goal is that if we can keep those attributes accurate you just need to look at the method and determine the complete retrial policy that it has -all the way down.
This means we need to add the attribute to all methods, and keep an eye on them to make sure that they don't go invalid/out of date after changes underneath.
It's a shame we can't automate this -but the need to have them does force us to audit the code. Like you say: we mustn't retry around a retry -but we must have a retry somewhere above every non-retried operation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1838:
HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377164858
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
Review comment:
even though I support this spelling, I'm afraid we need to use the US one. That avoids us having field bug reports about misspelt words
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org