You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/07/21 23:27:23 UTC
[GitHub] [hadoop] huaxiangsun opened a new pull request, #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
huaxiangsun opened a new pull request, #4608:
URL: https://github.com/apache/hadoop/pull/4608
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
processDeleteOnExit() is overiden in S3AFilesystem, it skips exist() check and delete objects without checking if FileSystem is closed.
### How was this patch tested?
A new unitest case is added. And all unittest cases under hadoop-tools/hadoop-aws passed.
mvn -Dparallel-tests clean test
Did S3A Integration tests against us-west-2 region and there are a few failures/errors. Run the trunk code without the patch, there are same errors/failures. The errors/failures are not caused by the patch, probably due to misconfiguration ( I could not figure out)
mvn -Dparallel-tests clean verify
The result is
`
Tests | Errors | Failures | Skipped | Success Rate | Time
-- | -- | -- | -- | -- | --
1252 | 6 | 1 | 270 | 77.875% | 3,627.473
`
The errors are
`
org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob#testCommonCrawlLookup[1] + [ Detail ] | 0.324
| s3a://hbase-test-data/fork-0001/test: getFileStatus on s3a://hbase-test-data/fork-0001/test: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: XJ3DCCR6Q7SXTJDW; S3 Extended Request ID: fRmP3m1lThWxhj3s9VkSNEtuBz1JeBWYw65aRajrSg/H7IN+muB7d8PavSeqJ2urvLZtguTbnlc=; Proxy: null), S3 Extended Request ID: fRmP3m1lThWxhj3s9VkSNEtuBz1JeBWYw65aRajrSg/H7IN+muB7d8PavSeqJ2urvLZtguTbnlc=:InvalidAccessKeyId |
| |
| testJobSubmissionCollectsTokens[1] + [ Detail ] | 0.329
| s3a://hbase-test-data/fork-0001/test: getFileStatus on s3a://hbase-test-data/fork-0001/test: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: XJ35WHK7X6EMP9B6; S3 Extended Request ID: rtWEsDYcGqNiaoKy2D5EQQqN+O7MbYe1bYbiSmkF+FOz9/wb6+t+dQooqj7ppCSCZMBgC3PeEw4=; Proxy: null), S3 Extended Request ID: rtWEsDYcGqNiaoKy2D5EQQqN+O7MbYe1bYbiSmkF+FOz9/wb6+t+dQooqj7ppCSCZMBgC3PeEw4=:InvalidAccessKeyId
`
`
org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion#testBlankRegionTriggersSDKResolution + [ Detail ] | 2.817
| [Client region name] expected:<"[mars-north]-2"> but was:<"[us-west]-2">
`
and
`
org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials#estSTS
| : request session credentials: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Cannot call GetSessionToken with session credentials (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: d935696d-7d98-4a1c-825f-22bf3d28ed9d; Proxy: null):AccessDenied
`
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1203263503
Thanks @steveloughran for all the helps. I pushed a new patch, which I hope addresses your comments.
1. Undo the changes in FileSystem.java, code is duplicated in S3AFileSystem.
2. Incorporated some of your changes in [HADOOP-13611](https://issues.apache.org/jira/browse/HADOOP-13611)
3. Added a new s3a integration test case for deleteOnExit.
I have run all integration test under hadoop-aws. Here are the results.
<div id="breadcrumbs" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; color: rgb(0, 0, 0); font-family: Times; font-size: medium; font-variant-ligatures: normal; orphans: 2; widows: 2; text-decoration-thickness: initial;"><div class="xleft"><span id="publishDate">Last Published: 2022-08-02</span> | <span id="projectVersion">Version: 3.4.0-SNAPSHOT</span></div><div class="xright"></div><div class="clear"><hr></div></div><div id="leftColumn" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; color: rgb(0, 0, 0); font-f
amily: Times; font-size: medium; font-variant-ligatures: normal; orphans: 2; widows: 2; text-decoration-thickness: initial;"><div id="navcolumn"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy" style="color: var(--accent-color) !important; text-decoration: underline !important; -apple-color-filter: none;"></a></div></div><div id="bodyColumn" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; color: rgb(0, 0, 0); font-family: Times; font-size: medium; font-variant-ligatures: normal; orphans: 2; widows: 2; text-decoration-thickness: initial;"><div id="contentBox"><section><h2><a name="Failsafe_Report" style="color: rgb(158, 75, 47); text-decoration: underline !important;"></a>Failsafe Report</h2></section><section><h2><a name="S
ummary" style="color: rgb(158, 75, 47); text-decoration: underline !important;"></a>Summary</h2><a name="Summary" style="color: rgb(158, 75, 47); text-decoration: underline !important;"></a><p>[<a style="color: rgb(158, 75, 47); text-decoration: underline !important;">Summary</a>] [<a style="color: rgb(158, 75, 47); text-decoration: underline !important;">Package List</a>] [<a style="color: rgb(158, 75, 47); text-decoration: underline !important;">Test Cases</a>]</p><br>
Tests | Errors | Failures | Skipped | Success Rate | Time
-- | -- | -- | -- | -- | --
1253 | 7 | 0 | 270 | 77.893% | 3,301.515
</section></div></div>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1193434292
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 54s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 33s | | trunk passed |
| +1 :green_heart: | compile | 30m 25s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 26m 29s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 5m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 56s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 48s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | | the patch passed |
| +1 :green_heart: | compile | 27m 36s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 27m 36s | | the patch passed |
| +1 :green_heart: | compile | 26m 25s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 26m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 54s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 30s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 17s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 45s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 44s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 34s | | The patch does not generate ASF License warnings. |
| | | 287m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4608 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 355061410a40 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / db475edeed8ba4e872f4ced8db44fce3eb6b3d35 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/3/testReport/ |
| Max. process+thread count | 1379 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1206668465
@steveloughran Sorry to bother you while you are busy, just a gentle reminder for the patch I posted couple days ago. Appreciate that you can review the patch when you get chance, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1209917783
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 2s | | trunk passed |
| +1 :green_heart: | compile | 1m 4s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 59s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not generate ASF License warnings. |
| | | 116m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4608 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 02e1a93d5d2e 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fb2ae77ed7a2f598dcd24cae9d43c944045bb81f |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/8/testReport/ |
| Max. process+thread count | 607 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
steveloughran commented on code in PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#discussion_r940572222
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.s3a;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Test deleteOnExit for S3A.
+ * The following cases for deleteOnExit are tested:
+ * 1. A nonexist file, which is added to deleteOnExit set.
+ * 2. An existing file
+ * 3. A file is added to deleteOnExist set first, then created.
+ * 4. A directory with some files under it.
+ */
+public class ITestS3ADeleteOnExit extends AbstractS3ATestBase {
+
+ private static final String PARENT_DIR_PATH_STR = "testDeleteOnExitDir";
+ private static final String NON_EXIST_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/nonExistFile";
+ private static final String INORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/inOrderFile";
+ private static final String OUT_OF_ORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/outOfOrderFile";
+ private static final String SUBDIR_PATH_STR =
+ PARENT_DIR_PATH_STR + "/subDir";
+ private static final String FILE_UNDER_SUBDIR_PATH_STR =
+ SUBDIR_PATH_STR + "/subDirFile";
+
+ @Test
+ public void testDeleteOnExit() throws Exception {
+ FileSystem fs = getFileSystem();
+
+ // Get a new filesystem object which is same as fs.
+ FileSystem s3aFs = new S3AFileSystem();
+ s3aFs.initialize(fs.getUri(), fs.getConf());
+ Path nonExistFilePath = path(NON_EXIST_FILE_PATH_STR);
+ Path inOrderFilePath = path(INORDER_FILE_PATH_STR);
+ Path outOfOrderFilePath = path(OUT_OF_ORDER_FILE_PATH_STR);
+ Path subDirPath = path(SUBDIR_PATH_STR);
+ Path fileUnderSubDirPath = path(FILE_UNDER_SUBDIR_PATH_STR);
+ // 1. set up the test directory.
+ Path dir = path("testDeleteOnExitDir");
+ s3aFs.mkdirs(dir);
+
+ // 2. Add a nonexisting file to DeleteOnExit set.
+ s3aFs.deleteOnExit(nonExistFilePath);
+ ContractTestUtils.assertPathDoesNotExist(s3aFs,
+ "File " + NON_EXIST_FILE_PATH_STR + " should not exist", nonExistFilePath);
+
+ // 3. create a file and then add it to DeleteOnExit set.
+ FSDataOutputStream stream = s3aFs.create(inOrderFilePath, true);
+ byte[] data = ContractTestUtils.dataset(16, 'a', 26);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + INORDER_FILE_PATH_STR + " should exist", inOrderFilePath);
+
+ s3aFs.deleteOnExit(inOrderFilePath);
+
+ // 4. add a path to DeleteOnExit set first, then create it.
+ s3aFs.deleteOnExit(outOfOrderFilePath);
+ stream = s3aFs.create(outOfOrderFilePath, true);
Review Comment:
again, ContractTestUtils.createFile
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.s3a;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Test deleteOnExit for S3A.
+ * The following cases for deleteOnExit are tested:
+ * 1. A nonexist file, which is added to deleteOnExit set.
+ * 2. An existing file
+ * 3. A file is added to deleteOnExist set first, then created.
+ * 4. A directory with some files under it.
+ */
+public class ITestS3ADeleteOnExit extends AbstractS3ATestBase {
+
+ private static final String PARENT_DIR_PATH_STR = "testDeleteOnExitDir";
+ private static final String NON_EXIST_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/nonExistFile";
+ private static final String INORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/inOrderFile";
+ private static final String OUT_OF_ORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/outOfOrderFile";
+ private static final String SUBDIR_PATH_STR =
+ PARENT_DIR_PATH_STR + "/subDir";
+ private static final String FILE_UNDER_SUBDIR_PATH_STR =
+ SUBDIR_PATH_STR + "/subDirFile";
+
+ @Test
+ public void testDeleteOnExit() throws Exception {
+ FileSystem fs = getFileSystem();
+
+ // Get a new filesystem object which is same as fs.
+ FileSystem s3aFs = new S3AFileSystem();
+ s3aFs.initialize(fs.getUri(), fs.getConf());
+ Path nonExistFilePath = path(NON_EXIST_FILE_PATH_STR);
+ Path inOrderFilePath = path(INORDER_FILE_PATH_STR);
+ Path outOfOrderFilePath = path(OUT_OF_ORDER_FILE_PATH_STR);
+ Path subDirPath = path(SUBDIR_PATH_STR);
+ Path fileUnderSubDirPath = path(FILE_UNDER_SUBDIR_PATH_STR);
+ // 1. set up the test directory.
+ Path dir = path("testDeleteOnExitDir");
+ s3aFs.mkdirs(dir);
+
+ // 2. Add a nonexisting file to DeleteOnExit set.
+ s3aFs.deleteOnExit(nonExistFilePath);
+ ContractTestUtils.assertPathDoesNotExist(s3aFs,
+ "File " + NON_EXIST_FILE_PATH_STR + " should not exist", nonExistFilePath);
+
+ // 3. create a file and then add it to DeleteOnExit set.
+ FSDataOutputStream stream = s3aFs.create(inOrderFilePath, true);
+ byte[] data = ContractTestUtils.dataset(16, 'a', 26);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + INORDER_FILE_PATH_STR + " should exist", inOrderFilePath);
+
+ s3aFs.deleteOnExit(inOrderFilePath);
+
+ // 4. add a path to DeleteOnExit set first, then create it.
+ s3aFs.deleteOnExit(outOfOrderFilePath);
+ stream = s3aFs.create(outOfOrderFilePath, true);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + OUT_OF_ORDER_FILE_PATH_STR + " should exist", outOfOrderFilePath);
+
+ // 5. create a subdirectory, a file under it, and add subdirectory DeleteOnExit set.
+ s3aFs.mkdirs(subDirPath);
+ s3aFs.deleteOnExit(subDirPath);
+
+ stream = s3aFs.create(fileUnderSubDirPath, true);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
Review Comment:
you are using this enough a static import makes sense
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.s3a;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Test deleteOnExit for S3A.
+ * The following cases for deleteOnExit are tested:
+ * 1. A nonexist file, which is added to deleteOnExit set.
+ * 2. An existing file
+ * 3. A file is added to deleteOnExist set first, then created.
+ * 4. A directory with some files under it.
+ */
+public class ITestS3ADeleteOnExit extends AbstractS3ATestBase {
+
+ private static final String PARENT_DIR_PATH_STR = "testDeleteOnExitDir";
+ private static final String NON_EXIST_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/nonExistFile";
+ private static final String INORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/inOrderFile";
+ private static final String OUT_OF_ORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/outOfOrderFile";
+ private static final String SUBDIR_PATH_STR =
+ PARENT_DIR_PATH_STR + "/subDir";
+ private static final String FILE_UNDER_SUBDIR_PATH_STR =
+ SUBDIR_PATH_STR + "/subDirFile";
+
+ @Test
+ public void testDeleteOnExit() throws Exception {
+ FileSystem fs = getFileSystem();
+
+ // Get a new filesystem object which is same as fs.
+ FileSystem s3aFs = new S3AFileSystem();
+ s3aFs.initialize(fs.getUri(), fs.getConf());
+ Path nonExistFilePath = path(NON_EXIST_FILE_PATH_STR);
+ Path inOrderFilePath = path(INORDER_FILE_PATH_STR);
+ Path outOfOrderFilePath = path(OUT_OF_ORDER_FILE_PATH_STR);
+ Path subDirPath = path(SUBDIR_PATH_STR);
+ Path fileUnderSubDirPath = path(FILE_UNDER_SUBDIR_PATH_STR);
+ // 1. set up the test directory.
+ Path dir = path("testDeleteOnExitDir");
+ s3aFs.mkdirs(dir);
+
+ // 2. Add a nonexisting file to DeleteOnExit set.
+ s3aFs.deleteOnExit(nonExistFilePath);
+ ContractTestUtils.assertPathDoesNotExist(s3aFs,
+ "File " + NON_EXIST_FILE_PATH_STR + " should not exist", nonExistFilePath);
+
+ // 3. create a file and then add it to DeleteOnExit set.
+ FSDataOutputStream stream = s3aFs.create(inOrderFilePath, true);
Review Comment:
use ContractTestUtils.createFile here
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.s3a;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Test deleteOnExit for S3A.
+ * The following cases for deleteOnExit are tested:
+ * 1. A nonexist file, which is added to deleteOnExit set.
+ * 2. An existing file
+ * 3. A file is added to deleteOnExist set first, then created.
+ * 4. A directory with some files under it.
+ */
+public class ITestS3ADeleteOnExit extends AbstractS3ATestBase {
+
+ private static final String PARENT_DIR_PATH_STR = "testDeleteOnExitDir";
+ private static final String NON_EXIST_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/nonExistFile";
+ private static final String INORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/inOrderFile";
+ private static final String OUT_OF_ORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/outOfOrderFile";
+ private static final String SUBDIR_PATH_STR =
+ PARENT_DIR_PATH_STR + "/subDir";
+ private static final String FILE_UNDER_SUBDIR_PATH_STR =
+ SUBDIR_PATH_STR + "/subDirFile";
+
+ @Test
+ public void testDeleteOnExit() throws Exception {
+ FileSystem fs = getFileSystem();
+
+ // Get a new filesystem object which is same as fs.
+ FileSystem s3aFs = new S3AFileSystem();
+ s3aFs.initialize(fs.getUri(), fs.getConf());
+ Path nonExistFilePath = path(NON_EXIST_FILE_PATH_STR);
+ Path inOrderFilePath = path(INORDER_FILE_PATH_STR);
+ Path outOfOrderFilePath = path(OUT_OF_ORDER_FILE_PATH_STR);
+ Path subDirPath = path(SUBDIR_PATH_STR);
+ Path fileUnderSubDirPath = path(FILE_UNDER_SUBDIR_PATH_STR);
+ // 1. set up the test directory.
+ Path dir = path("testDeleteOnExitDir");
+ s3aFs.mkdirs(dir);
+
+ // 2. Add a nonexisting file to DeleteOnExit set.
+ s3aFs.deleteOnExit(nonExistFilePath);
+ ContractTestUtils.assertPathDoesNotExist(s3aFs,
+ "File " + NON_EXIST_FILE_PATH_STR + " should not exist", nonExistFilePath);
+
+ // 3. create a file and then add it to DeleteOnExit set.
+ FSDataOutputStream stream = s3aFs.create(inOrderFilePath, true);
+ byte[] data = ContractTestUtils.dataset(16, 'a', 26);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + INORDER_FILE_PATH_STR + " should exist", inOrderFilePath);
+
+ s3aFs.deleteOnExit(inOrderFilePath);
+
+ // 4. add a path to DeleteOnExit set first, then create it.
+ s3aFs.deleteOnExit(outOfOrderFilePath);
+ stream = s3aFs.create(outOfOrderFilePath, true);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + OUT_OF_ORDER_FILE_PATH_STR + " should exist", outOfOrderFilePath);
+
+ // 5. create a subdirectory, a file under it, and add subdirectory DeleteOnExit set.
+ s3aFs.mkdirs(subDirPath);
+ s3aFs.deleteOnExit(subDirPath);
+
+ stream = s3aFs.create(fileUnderSubDirPath, true);
+ try {
Review Comment:
same sing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1192684231
Will address the checkstyle issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on a diff in pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on code in PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#discussion_r941726247
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.s3a;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Test deleteOnExit for S3A.
+ * The following cases for deleteOnExit are tested:
+ * 1. A nonexist file, which is added to deleteOnExit set.
+ * 2. An existing file
+ * 3. A file is added to deleteOnExist set first, then created.
+ * 4. A directory with some files under it.
+ */
+public class ITestS3ADeleteOnExit extends AbstractS3ATestBase {
+
+ private static final String PARENT_DIR_PATH_STR = "testDeleteOnExitDir";
+ private static final String NON_EXIST_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/nonExistFile";
+ private static final String INORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/inOrderFile";
+ private static final String OUT_OF_ORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/outOfOrderFile";
+ private static final String SUBDIR_PATH_STR =
+ PARENT_DIR_PATH_STR + "/subDir";
+ private static final String FILE_UNDER_SUBDIR_PATH_STR =
+ SUBDIR_PATH_STR + "/subDirFile";
+
+ @Test
+ public void testDeleteOnExit() throws Exception {
+ FileSystem fs = getFileSystem();
+
+ // Get a new filesystem object which is same as fs.
+ FileSystem s3aFs = new S3AFileSystem();
+ s3aFs.initialize(fs.getUri(), fs.getConf());
+ Path nonExistFilePath = path(NON_EXIST_FILE_PATH_STR);
+ Path inOrderFilePath = path(INORDER_FILE_PATH_STR);
+ Path outOfOrderFilePath = path(OUT_OF_ORDER_FILE_PATH_STR);
+ Path subDirPath = path(SUBDIR_PATH_STR);
+ Path fileUnderSubDirPath = path(FILE_UNDER_SUBDIR_PATH_STR);
+ // 1. set up the test directory.
+ Path dir = path("testDeleteOnExitDir");
+ s3aFs.mkdirs(dir);
+
+ // 2. Add a nonexisting file to DeleteOnExit set.
+ s3aFs.deleteOnExit(nonExistFilePath);
+ ContractTestUtils.assertPathDoesNotExist(s3aFs,
+ "File " + NON_EXIST_FILE_PATH_STR + " should not exist", nonExistFilePath);
+
+ // 3. create a file and then add it to DeleteOnExit set.
+ FSDataOutputStream stream = s3aFs.create(inOrderFilePath, true);
+ byte[] data = ContractTestUtils.dataset(16, 'a', 26);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + INORDER_FILE_PATH_STR + " should exist", inOrderFilePath);
+
+ s3aFs.deleteOnExit(inOrderFilePath);
+
+ // 4. add a path to DeleteOnExit set first, then create it.
+ s3aFs.deleteOnExit(outOfOrderFilePath);
+ stream = s3aFs.create(outOfOrderFilePath, true);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + OUT_OF_ORDER_FILE_PATH_STR + " should exist", outOfOrderFilePath);
+
+ // 5. create a subdirectory, a file under it, and add subdirectory DeleteOnExit set.
+ s3aFs.mkdirs(subDirPath);
+ s3aFs.deleteOnExit(subDirPath);
+
+ stream = s3aFs.create(fileUnderSubDirPath, true);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
Review Comment:
I found that I do not need ContractTestUtils#assertPathExists or ContractTestUtils#assertPathDoesNotExist. I can use AbstractFSContractTestBase#assertPathExists and assertPathDoesNotExist instead. Upload the new patch based on your feedback, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1209793570
I also run the new IT test without change in S3AFileSystem, it failed as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1199117000
yes; look at some of the other ITest cases in the s3a code.
if you aren't set up to run the hadoop-aws test suite, get set up to do that, as we require anyone submitting a pr to run the tests themselves first. getting them to work in your IDE is best for debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
steveloughran commented on code in PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#discussion_r932130167
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.ArgumentMatchers.argThat;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Date;
+
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.GetObjectMetadataRequest;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
Review Comment:
nit: all non-static org apache imports should go in their own block below this one
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##########
@@ -3800,6 +3819,32 @@ UploadResult waitForUploadCompletion(String key, UploadInfo uploadInfo)
}
}
+
+ /**
+ * Delete all paths that were marked as delete-on-exit. This recursively
+ * deletes all files and directories in the specified paths. It does not
+ * check if file exists and filesystem is closed.
+ *
+ * The time to process this operation is {@code O(paths)}, with the actual
+ * time dependent on the time for existence and deletion operations to
+ * complete, successfully or not.
+ */
+ @Override
+ protected void processDeleteOnExit() {
+ Set<Path> deleteOnExit = getDeleteOnExitSet();
+ synchronized (deleteOnExit) {
+ for (Iterator<Path> iter = deleteOnExit.iterator(); iter.hasNext();) {
+ Path path = iter.next();
+ try {
+ deleteWithoutCloseCheck(path, true);
+ } catch (IOException e) {
+ LOG.info("Ignoring failure to deleteOnExit for path {}, exception {}", path, e);
Review Comment:
could be very noisy if there are connectivity problems. how about logging e.toString at info, full stack at debug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun closed pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun closed pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
URL: https://github.com/apache/hadoop/pull/4608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1208499534
yes, sorry, been neglecting this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
steveloughran commented on code in PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#discussion_r940572367
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.s3a;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Test deleteOnExit for S3A.
+ * The following cases for deleteOnExit are tested:
+ * 1. A nonexist file, which is added to deleteOnExit set.
+ * 2. An existing file
+ * 3. A file is added to deleteOnExist set first, then created.
+ * 4. A directory with some files under it.
+ */
+public class ITestS3ADeleteOnExit extends AbstractS3ATestBase {
+
+ private static final String PARENT_DIR_PATH_STR = "testDeleteOnExitDir";
+ private static final String NON_EXIST_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/nonExistFile";
+ private static final String INORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/inOrderFile";
+ private static final String OUT_OF_ORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/outOfOrderFile";
+ private static final String SUBDIR_PATH_STR =
+ PARENT_DIR_PATH_STR + "/subDir";
+ private static final String FILE_UNDER_SUBDIR_PATH_STR =
+ SUBDIR_PATH_STR + "/subDirFile";
+
+ @Test
+ public void testDeleteOnExit() throws Exception {
+ FileSystem fs = getFileSystem();
+
+ // Get a new filesystem object which is same as fs.
+ FileSystem s3aFs = new S3AFileSystem();
+ s3aFs.initialize(fs.getUri(), fs.getConf());
+ Path nonExistFilePath = path(NON_EXIST_FILE_PATH_STR);
+ Path inOrderFilePath = path(INORDER_FILE_PATH_STR);
+ Path outOfOrderFilePath = path(OUT_OF_ORDER_FILE_PATH_STR);
+ Path subDirPath = path(SUBDIR_PATH_STR);
+ Path fileUnderSubDirPath = path(FILE_UNDER_SUBDIR_PATH_STR);
+ // 1. set up the test directory.
+ Path dir = path("testDeleteOnExitDir");
+ s3aFs.mkdirs(dir);
+
+ // 2. Add a nonexisting file to DeleteOnExit set.
+ s3aFs.deleteOnExit(nonExistFilePath);
+ ContractTestUtils.assertPathDoesNotExist(s3aFs,
+ "File " + NON_EXIST_FILE_PATH_STR + " should not exist", nonExistFilePath);
+
+ // 3. create a file and then add it to DeleteOnExit set.
+ FSDataOutputStream stream = s3aFs.create(inOrderFilePath, true);
+ byte[] data = ContractTestUtils.dataset(16, 'a', 26);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + INORDER_FILE_PATH_STR + " should exist", inOrderFilePath);
+
+ s3aFs.deleteOnExit(inOrderFilePath);
+
+ // 4. add a path to DeleteOnExit set first, then create it.
+ s3aFs.deleteOnExit(outOfOrderFilePath);
+ stream = s3aFs.create(outOfOrderFilePath, true);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + OUT_OF_ORDER_FILE_PATH_STR + " should exist", outOfOrderFilePath);
+
+ // 5. create a subdirectory, a file under it, and add subdirectory DeleteOnExit set.
+ s3aFs.mkdirs(subDirPath);
+ s3aFs.deleteOnExit(subDirPath);
+
+ stream = s3aFs.create(fileUnderSubDirPath, true);
+ try {
Review Comment:
same thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on a diff in pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on code in PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#discussion_r932404542
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.ArgumentMatchers.argThat;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Date;
+
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.GetObjectMetadataRequest;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
Review Comment:
Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on a diff in pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on code in PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#discussion_r941552836
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.s3a;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Test deleteOnExit for S3A.
+ * The following cases for deleteOnExit are tested:
+ * 1. A nonexist file, which is added to deleteOnExit set.
+ * 2. An existing file
+ * 3. A file is added to deleteOnExist set first, then created.
+ * 4. A directory with some files under it.
+ */
+public class ITestS3ADeleteOnExit extends AbstractS3ATestBase {
+
+ private static final String PARENT_DIR_PATH_STR = "testDeleteOnExitDir";
+ private static final String NON_EXIST_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/nonExistFile";
+ private static final String INORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/inOrderFile";
+ private static final String OUT_OF_ORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/outOfOrderFile";
+ private static final String SUBDIR_PATH_STR =
+ PARENT_DIR_PATH_STR + "/subDir";
+ private static final String FILE_UNDER_SUBDIR_PATH_STR =
+ SUBDIR_PATH_STR + "/subDirFile";
+
+ @Test
+ public void testDeleteOnExit() throws Exception {
+ FileSystem fs = getFileSystem();
+
+ // Get a new filesystem object which is same as fs.
+ FileSystem s3aFs = new S3AFileSystem();
+ s3aFs.initialize(fs.getUri(), fs.getConf());
+ Path nonExistFilePath = path(NON_EXIST_FILE_PATH_STR);
+ Path inOrderFilePath = path(INORDER_FILE_PATH_STR);
+ Path outOfOrderFilePath = path(OUT_OF_ORDER_FILE_PATH_STR);
+ Path subDirPath = path(SUBDIR_PATH_STR);
+ Path fileUnderSubDirPath = path(FILE_UNDER_SUBDIR_PATH_STR);
+ // 1. set up the test directory.
+ Path dir = path("testDeleteOnExitDir");
+ s3aFs.mkdirs(dir);
+
+ // 2. Add a nonexisting file to DeleteOnExit set.
+ s3aFs.deleteOnExit(nonExistFilePath);
+ ContractTestUtils.assertPathDoesNotExist(s3aFs,
+ "File " + NON_EXIST_FILE_PATH_STR + " should not exist", nonExistFilePath);
+
+ // 3. create a file and then add it to DeleteOnExit set.
+ FSDataOutputStream stream = s3aFs.create(inOrderFilePath, true);
+ byte[] data = ContractTestUtils.dataset(16, 'a', 26);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + INORDER_FILE_PATH_STR + " should exist", inOrderFilePath);
+
+ s3aFs.deleteOnExit(inOrderFilePath);
+
+ // 4. add a path to DeleteOnExit set first, then create it.
+ s3aFs.deleteOnExit(outOfOrderFilePath);
+ stream = s3aFs.create(outOfOrderFilePath, true);
Review Comment:
Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1198721463
> code in s3afs and the tests all look good.
>
> But i think the changes should be restricted to s3a fs, even if duplicates a bit of the superclass.
>
> That whole section in the javadocs at the top of the Filesystem. class explains why; you've already seen some of those test values but it's and external implementations which we don't have control of. Adding anything is making a commitment to preserve a new public API forever.
Thanks! Points well taken.
>
> I will be happier if you were just do it all in S3AFileSystem.
>
> override deleteOnExit(Path f) and
>
> 1. skip the exists check because it doesn't do the right thing if you call the method while writing a file, because the file isn't visible until close. saves HEAD/List probes too
> 2. save the list to a set local to s3a fs. you could make its getter protected so that mock test can access it.
Got it, for s3a fs, the deleteOnExit set is going to be in s3a.
>
> now, what about an integration test too?
Will try to add an IT.
>
> add a test case which creates a new fs in the test case (keeping the normal getFileSystem() fs for assertions)
>
> * adds a file which doesn't exist
> * adds a file which doesn't exist, then create it
> * adds a file which does exist
> * add a directory path
> * close the fs, use ContractTestUtils methods to assert the files and dirs are gone.
>
> seem good?
Sounds good to me, thanks for the feedbacks. May come back to you for test case, going to check it out first by myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1197071839
Hi @steveloughran, can you take a look at the patch? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340. deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1212494590
Thanks @steveloughran for the review, really nice comments!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on a diff in pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on code in PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#discussion_r941713106
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.s3a;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Test deleteOnExit for S3A.
+ * The following cases for deleteOnExit are tested:
+ * 1. A nonexist file, which is added to deleteOnExit set.
+ * 2. An existing file
+ * 3. A file is added to deleteOnExist set first, then created.
+ * 4. A directory with some files under it.
+ */
+public class ITestS3ADeleteOnExit extends AbstractS3ATestBase {
+
+ private static final String PARENT_DIR_PATH_STR = "testDeleteOnExitDir";
+ private static final String NON_EXIST_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/nonExistFile";
+ private static final String INORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/inOrderFile";
+ private static final String OUT_OF_ORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/outOfOrderFile";
+ private static final String SUBDIR_PATH_STR =
+ PARENT_DIR_PATH_STR + "/subDir";
+ private static final String FILE_UNDER_SUBDIR_PATH_STR =
+ SUBDIR_PATH_STR + "/subDirFile";
+
+ @Test
+ public void testDeleteOnExit() throws Exception {
+ FileSystem fs = getFileSystem();
+
+ // Get a new filesystem object which is same as fs.
+ FileSystem s3aFs = new S3AFileSystem();
+ s3aFs.initialize(fs.getUri(), fs.getConf());
+ Path nonExistFilePath = path(NON_EXIST_FILE_PATH_STR);
+ Path inOrderFilePath = path(INORDER_FILE_PATH_STR);
+ Path outOfOrderFilePath = path(OUT_OF_ORDER_FILE_PATH_STR);
+ Path subDirPath = path(SUBDIR_PATH_STR);
+ Path fileUnderSubDirPath = path(FILE_UNDER_SUBDIR_PATH_STR);
+ // 1. set up the test directory.
+ Path dir = path("testDeleteOnExitDir");
+ s3aFs.mkdirs(dir);
+
+ // 2. Add a nonexisting file to DeleteOnExit set.
+ s3aFs.deleteOnExit(nonExistFilePath);
+ ContractTestUtils.assertPathDoesNotExist(s3aFs,
+ "File " + NON_EXIST_FILE_PATH_STR + " should not exist", nonExistFilePath);
+
+ // 3. create a file and then add it to DeleteOnExit set.
+ FSDataOutputStream stream = s3aFs.create(inOrderFilePath, true);
+ byte[] data = ContractTestUtils.dataset(16, 'a', 26);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + INORDER_FILE_PATH_STR + " should exist", inOrderFilePath);
+
+ s3aFs.deleteOnExit(inOrderFilePath);
+
+ // 4. add a path to DeleteOnExit set first, then create it.
+ s3aFs.deleteOnExit(outOfOrderFilePath);
+ stream = s3aFs.create(outOfOrderFilePath, true);
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1192997860
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 32s | | trunk passed |
| +1 :green_heart: | compile | 25m 20s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 22m 3s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 23s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 5s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 41s | | the patch passed |
| +1 :green_heart: | compile | 24m 32s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 32s | | the patch passed |
| +1 :green_heart: | compile | 21m 57s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 13s | | the patch passed |
| -1 :x: | javadoc | 1m 24s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/2/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | javadoc | 2m 5s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 40s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 18m 36s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 0s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 18s | | The patch does not generate ASF License warnings. |
| | | 251m 11s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.TestHarFileSystem |
| | hadoop.fs.TestFilterFileSystem |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4608 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0771c43e5080 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d5947073c7487de2ea8696d49e6fc41785dcc3be |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/2/testReport/ |
| Max. process+thread count | 1278 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1203268821
I also run all unitest cases under hadoop-aws. They passed.
INFO] Results:
[INFO]
[WARNING] Tests run: 407, Failures: 0, Errors: 0, Skipped: 4
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:37 min
[INFO] Finished at: 2022-08-02T10:29:52-07:00
[INFO] ------------------------------------------------------------------------
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1203333343
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 2s | | trunk passed |
| +1 :green_heart: | compile | 1m 6s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 59s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 18s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 25s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 20s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not generate ASF License warnings. |
| | | 117m 46s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4608 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d752737fdab7 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 74f042d335202da1ee13c740cc1bbd14ede630b1 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/4/testReport/ |
| Max. process+thread count | 531 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1203340397
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 32s | | trunk passed |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 25s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 0s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 98m 55s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4608 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3e6aca5c741f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 38cd6ad39ec3ecefcef4261c776778a35f58b9fc |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/5/testReport/ |
| Max. process+thread count | 563 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1209900584
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 13s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 29s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 98m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4608 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 98c795ea550b 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fb2ae77ed7a2f598dcd24cae9d43c944045bb81f |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/7/testReport/ |
| Max. process+thread count | 554 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1209789086
> all the production code is good; some minor changes to the tests and its ready for the next hadoop release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1192147355
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 50s | | trunk passed |
| +1 :green_heart: | compile | 25m 25s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 54s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 5m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 56s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 42s | | the patch passed |
| +1 :green_heart: | compile | 24m 29s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 29s | | the patch passed |
| +1 :green_heart: | compile | 21m 57s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 27s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 70 unchanged - 0 fixed = 73 total (was 70) |
| +1 :green_heart: | mvnsite | 3m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 17s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 7s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 38s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 37s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 8s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 20s | | The patch does not generate ASF License warnings. |
| | | 252m 55s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4608 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 422779c483bf 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6e4a99265d8935a2c67e47b5f5e3a364d87c4f77 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/1/testReport/ |
| Max. process+thread count | 3137 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on a diff in pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on code in PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#discussion_r932404324
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##########
@@ -3800,6 +3819,32 @@ UploadResult waitForUploadCompletion(String key, UploadInfo uploadInfo)
}
}
+
+ /**
+ * Delete all paths that were marked as delete-on-exit. This recursively
+ * deletes all files and directories in the specified paths. It does not
+ * check if file exists and filesystem is closed.
+ *
+ * The time to process this operation is {@code O(paths)}, with the actual
+ * time dependent on the time for existence and deletion operations to
+ * complete, successfully or not.
+ */
+ @Override
+ protected void processDeleteOnExit() {
+ Set<Path> deleteOnExit = getDeleteOnExitSet();
+ synchronized (deleteOnExit) {
+ for (Iterator<Path> iter = deleteOnExit.iterator(); iter.hasNext();) {
+ Path path = iter.next();
+ try {
+ deleteWithoutCloseCheck(path, true);
+ } catch (IOException e) {
+ LOG.info("Ignoring failure to deleteOnExit for path {}, exception {}", path, e);
Review Comment:
Makes sense, will change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1201491627
Thanks for info for ITtest for s3a. I did run IT for s3a before submitting the patch, but did not try to run in IDE yet. Let me try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #4608: HADOOP-18340. deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
steveloughran merged PR #4608:
URL: https://github.com/apache/hadoop/pull/4608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1209905685
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 57s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 47s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. |
| | | 107m 8s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4608 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 107e0a86a207 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fb2ae77ed7a2f598dcd24cae9d43c944045bb81f |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/6/testReport/ |
| Max. process+thread count | 597 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huaxiangsun commented on a diff in pull request #4608: HADOOP-18340 deleteOnExit does not work with S3AFileSystem
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on code in PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#discussion_r941729359
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.s3a;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Test deleteOnExit for S3A.
+ * The following cases for deleteOnExit are tested:
+ * 1. A nonexist file, which is added to deleteOnExit set.
+ * 2. An existing file
+ * 3. A file is added to deleteOnExist set first, then created.
+ * 4. A directory with some files under it.
+ */
+public class ITestS3ADeleteOnExit extends AbstractS3ATestBase {
+
+ private static final String PARENT_DIR_PATH_STR = "testDeleteOnExitDir";
+ private static final String NON_EXIST_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/nonExistFile";
+ private static final String INORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/inOrderFile";
+ private static final String OUT_OF_ORDER_FILE_PATH_STR =
+ PARENT_DIR_PATH_STR + "/outOfOrderFile";
+ private static final String SUBDIR_PATH_STR =
+ PARENT_DIR_PATH_STR + "/subDir";
+ private static final String FILE_UNDER_SUBDIR_PATH_STR =
+ SUBDIR_PATH_STR + "/subDirFile";
+
+ @Test
+ public void testDeleteOnExit() throws Exception {
+ FileSystem fs = getFileSystem();
+
+ // Get a new filesystem object which is same as fs.
+ FileSystem s3aFs = new S3AFileSystem();
+ s3aFs.initialize(fs.getUri(), fs.getConf());
+ Path nonExistFilePath = path(NON_EXIST_FILE_PATH_STR);
+ Path inOrderFilePath = path(INORDER_FILE_PATH_STR);
+ Path outOfOrderFilePath = path(OUT_OF_ORDER_FILE_PATH_STR);
+ Path subDirPath = path(SUBDIR_PATH_STR);
+ Path fileUnderSubDirPath = path(FILE_UNDER_SUBDIR_PATH_STR);
+ // 1. set up the test directory.
+ Path dir = path("testDeleteOnExitDir");
+ s3aFs.mkdirs(dir);
+
+ // 2. Add a nonexisting file to DeleteOnExit set.
+ s3aFs.deleteOnExit(nonExistFilePath);
+ ContractTestUtils.assertPathDoesNotExist(s3aFs,
+ "File " + NON_EXIST_FILE_PATH_STR + " should not exist", nonExistFilePath);
+
+ // 3. create a file and then add it to DeleteOnExit set.
+ FSDataOutputStream stream = s3aFs.create(inOrderFilePath, true);
+ byte[] data = ContractTestUtils.dataset(16, 'a', 26);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + INORDER_FILE_PATH_STR + " should exist", inOrderFilePath);
+
+ s3aFs.deleteOnExit(inOrderFilePath);
+
+ // 4. add a path to DeleteOnExit set first, then create it.
+ s3aFs.deleteOnExit(outOfOrderFilePath);
+ stream = s3aFs.create(outOfOrderFilePath, true);
+ try {
+ stream.write(data);
+ } finally {
+ IOUtils.closeStream(stream);
+ }
+
+ ContractTestUtils.assertPathExists(s3aFs,
+ "File " + OUT_OF_ORDER_FILE_PATH_STR + " should exist", outOfOrderFilePath);
+
+ // 5. create a subdirectory, a file under it, and add subdirectory DeleteOnExit set.
+ s3aFs.mkdirs(subDirPath);
+ s3aFs.deleteOnExit(subDirPath);
+
+ stream = s3aFs.create(fileUnderSubDirPath, true);
+ try {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org