You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/03/30 16:36:49 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #1924:
HADOOP-16877. S3A FS deleteOnExit to skip the exists check.
steveloughran opened a new pull request #1924: HADOOP-16877. S3A FS deleteOnExit to skip the exists check.
URL: https://github.com/apache/hadoop/pull/1924
* new override method S3AFileSystem.deleteOnExit() which skips the exists
check.
* FileSystem.processDeleteOnExit() skips exists checks too; relies on delete()
to do its work.
* make sure all the delete/cancel/process deleteOnExit operations consistently
qualify paths, especially the list of paths to delete
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1924: HADOOP-16877. S3A
FS deleteOnExit to skip the exists check.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1924: HADOOP-16877. S3A FS deleteOnExit to skip the exists check.
URL: https://github.com/apache/hadoop/pull/1924#issuecomment-606507397
And of course I didn't read the javadocs in FileSystem I added last month so should have expected the stack traces. [INFO]
[ERROR] Failures:
[ERROR] TestFileSystemCaching.testCancelDeleteOnExit:313
[ERROR] TestFileSystemCaching.testDeleteOnExit:259
[ERROR] TestFileSystemCaching.testDeleteOnExitFNF:278
Argument(s) are different! Wanted:
fileSystem.getFileStatus(/a);
-> at org.apache.hadoop.fs.TestFileSystemCaching.testDeleteOnExitFNF(TestFileSystemCaching.java:278)
Actual invocations have different arguments:
fileSystem.makeQualified(/a);
-> at org.apache.hadoop.fs.FilterFileSystem.makeQualified(FilterFileSystem.java:124)
fileSystem.getFileStatus(null);
-> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:460)
[ERROR] TestFileSystemCaching.testDeleteOnExitRemoved:296
[ERROR] TestFilterFileSystem.testFilterFileSystem:170 1 methods were not overridden correctly - see log
[ERROR] TestHarFileSystem.testInheritedMethodsImplemented:393 1 methods were not overridden correctly - see log
[ERROR] Errors:
[ERROR] TestChRootedFileSystem.testDeleteOnExitPathHandling:346 ? NullPointer
[INFO]
[ERROR] Tests run: 4401, Failures: 6, Errors: 1, Skipped: 255
[INFO]
Hmm. I think I might just rewind all changes to FileSystem and have S3A build up its own list of files to delete and then delete them at teardown. Avoids complexity and would let us go to submitting delete operations to the thread pool
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1924: HADOOP-16877. S3A
FS deleteOnExit to skip the exists check.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1924: HADOOP-16877. S3A FS deleteOnExit to skip the exists check.
URL: https://github.com/apache/hadoop/pull/1924#issuecomment-606109821
Note: I tried to allow FS instances to support parallel deletes for extra performance, but it all gets too complex, especially as the delete process is synchronized on the list of paths to delete; easy to cause problems on something which shouldn't be a critical path (unless something uses it in production a lot). What is key: no 404 creation in any registration of paths to delete
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1924: HADOOP-16877. S3A
FS deleteOnExit to skip the exists check.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1924: HADOOP-16877. S3A FS deleteOnExit to skip the exists check.
URL: https://github.com/apache/hadoop/pull/1924#issuecomment-606108449
testing s3 london w/ s3guard.
There are no explicit tests of deleteOnExit, but lots of implicit ones as things are always being scheduled for deletion. I suppose I should do one though...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1924: HADOOP-16877. S3A
FS deleteOnExit to skip the exists check.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1924: HADOOP-16877. S3A FS deleteOnExit to skip the exists check.
URL: https://github.com/apache/hadoop/pull/1924#issuecomment-606176330
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 14s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 5s | trunk passed |
| +1 :green_heart: | compile | 19m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 3m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 28s | trunk passed |
| +1 :green_heart: | shadedclient | 23m 8s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 26s | trunk passed |
| +0 :ok: | spotbugs | 1m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 11s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | the patch passed |
| +1 :green_heart: | compile | 17m 20s | the patch passed |
| +1 :green_heart: | javac | 17m 20s | the patch passed |
| +1 :green_heart: | checkstyle | 2m 47s | the patch passed |
| +1 :green_heart: | mvnsite | 2m 7s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 30s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 27s | the patch passed |
| +1 :green_heart: | findbugs | 3m 27s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 10m 9s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 15s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
| | | 131m 16s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.viewfs.TestChRootedFileSystem |
| | hadoop.fs.TestHarFileSystem |
| | hadoop.fs.TestFileSystemCaching |
| | hadoop.fs.TestFilterFileSystem |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1924/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1924 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ab2c176bdab5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 960c9eb |
| Default Java | 1.8.0_242 |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1924/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1924/1/testReport/ |
| Max. process+thread count | 2605 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1924/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org