You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/03/04 17:50:35 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #2742: HADOOP-16721. Improve S3A rename resilience
steveloughran opened a new pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742
s3a rename to support
fs.s3a.rename.raises.exceptions: raise exceptions on rename failures
fs.s3a.rename.reduced.probes: don't look for parent dir (LIST), just verify
it isn't a file.
The reduced probe not only saves money, it avoids race conditions
where one thread deleting a subdir can cause LIST <parent> to fail before
a dir marker is recreated.
Note:
* file:// rename() creates parent dirs, so this isn't too dangerous.
* tests will switch modes.
We could always just do the HEAD; topic for discussion. This patch: optional
Change-Id: Ic0f8a410b45fef14ff522cb5aa1ae2bc19c8eeee
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r590194822
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestlRenameDeleteRace.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonClientException;
+import org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.AbstractS3ATestBase;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.util.BlockingThreadPoolExecutorService;
+
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY;
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY_DELETE;
+import static org.apache.hadoop.fs.s3a.Constants.S3GUARD_METASTORE_NULL;
+import static org.apache.hadoop.fs.s3a.Constants.S3_METADATA_STORE_IMPL;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.submit;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * HADOOP-16721: race condition with delete and rename underneath the same destination
+ * directory.
+ * This test suite recreates the failure using semaphores to guarantee the failure
+ * condition is encountered -then verifies that the rename operation is successful.
+ */
+public class ITestlRenameDeleteRace extends AbstractS3ATestBase {
Review comment:
yeah. well spotted
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r590433369
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestlRenameDeleteRace.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonClientException;
+import org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.AbstractS3ATestBase;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.util.BlockingThreadPoolExecutorService;
+
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY;
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY_DELETE;
+import static org.apache.hadoop.fs.s3a.Constants.S3GUARD_METASTORE_NULL;
+import static org.apache.hadoop.fs.s3a.Constants.S3_METADATA_STORE_IMPL;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.submit;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * HADOOP-16721: race condition with delete and rename underneath the same destination
+ * directory.
+ * This test suite recreates the failure using semaphores to guarantee the failure
+ * condition is encountered -then verifies that the rename operation is successful.
+ */
+public class ITestlRenameDeleteRace extends AbstractS3ATestBase {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ITestlRenameDeleteRace.class);
+
+
+ /** Many threads for scale performance: {@value}. */
+ public static final int EXECUTOR_THREAD_COUNT = 2;
+
+ /**
+ * For submitting work.
+ */
+ private static final ListeningExecutorService EXECUTOR =
+ BlockingThreadPoolExecutorService.newInstance(
+ EXECUTOR_THREAD_COUNT,
+ EXECUTOR_THREAD_COUNT * 2,
+ 30, TimeUnit.SECONDS,
+ "test-operations");
+
+ @Override
+ protected Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+ String bucketName = getTestBucketName(conf);
+
+ removeBaseAndBucketOverrides(bucketName, conf,
+ S3_METADATA_STORE_IMPL,
+ DIRECTORY_MARKER_POLICY);
+ // use the keep policy to ensure that surplus markers exist
+ // to complicate failures
+ conf.set(DIRECTORY_MARKER_POLICY, DIRECTORY_MARKER_POLICY_DELETE);
+ conf.set(S3_METADATA_STORE_IMPL, S3GUARD_METASTORE_NULL);
+
+ return conf;
+ }
+
+ @Test
+ public void testDeleteRenameRaceCondition() throws Throwable {
+ describe("verify no race between delete and rename");
+ final S3AFileSystem fs = getFileSystem();
+ final Path path = path(getMethodName());
+ Path srcDir = new Path(path, "src");
+
+ // this dir must exist throughout the rename
+ Path destDir = new Path(path, "dest");
+ // this dir tree will be deleted in a thread which does not
+ // complete before the rename exists
+ Path destSubdir1 = new Path(destDir, "subdir1");
+ Path subfile1 = new Path(destSubdir1, "subfile1");
+
+ // this is the directory we want to copy over under the dest dir
+ Path srcSubdir2 = new Path(srcDir, "subdir2");
+ Path srcSubfile = new Path(srcSubdir2, "subfile2");
+ Path destSubdir2 = new Path(destDir, "subdir2");
+
+ // creates subfile1 and all parents
+ ContractTestUtils.touch(fs, subfile1);
+ assertIsDirectory(destDir);
+
+ // source subfile
+ ContractTestUtils.touch(fs, srcSubfile);
+
+ final BlockingFakeDirMarkerFS blockingFS
+ = new BlockingFakeDirMarkerFS();
+ blockingFS.initialize(fs.getUri(), fs.getConf());
+ // get the semaphore; this ensures that the next attempt to create
+ // a fake marker blocks
+ try {
+ blockingFS.blockBeforCreatingMarker.acquire();
+ final CompletableFuture<Path> future = submit(EXECUTOR, () -> {
+ LOG.info("deleting {}", destSubdir1);
+ blockingFS.delete(destSubdir1, true);
+ return destSubdir1;
+ });
+
+ // wait for the delete to complete the deletion phase
+ blockingFS.signalCreatingFakeParentDirectory.acquire();
+
+ // there is now no destination directory
+ assertPathDoesNotExist("should have been implicitly deleted", destDir);
+
+ try {
+ // Now attempt the rename in the normal FS.
+ LOG.info("renaming {} to {}", srcSubdir2, destSubdir2);
+ Assertions.assertThat(fs.rename(srcSubdir2, destSubdir2))
+ .describedAs("rename(%s, %s)", srcSubdir2, destSubdir2)
+ .isTrue();
+ } finally {
+ blockingFS.blockBeforCreatingMarker.release();
+ }
+
+ // now let the delete complete
+ LOG.info("Waiting for delete {} to finish", destSubdir1);
+ waitForCompletion(future);
+ assertPathExists("must now exist", destDir);
+ assertPathExists("must now exist", new Path(destSubdir2, "subfile2"));
+ assertPathDoesNotExist("Src dir deleted", srcSubdir2);
+
+ } finally {
+ cleanupWithLogger(LOG, blockingFS);
+ }
+
+ }
+
+ /**
+ * Subclass of S3A FS whose execution of maybeCreateFakeParentDirectory
+ * can be choreographed with another thread so as to reliably
+ * create the delete/rename race condition.
+ */
+ private final class BlockingFakeDirMarkerFS extends S3AFileSystem {
+
+ /**
+ * Block for entry into maybeCreateFakeParentDirectory(); will be released
+ * then.
+ */
+ private final Semaphore signalCreatingFakeParentDirectory = new Semaphore(
+ 1);
+
+ /**
+ * Semaphore to acquire before the marker can be listed/created.
+ */
+ private final Semaphore blockBeforCreatingMarker = new Semaphore(1);
Review comment:
should be blockBefor**e**CreatingMarker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#issuecomment-790807398
Testing with -Dparallel-tests -DtestsThreadCount=8 -Dmarkers=keep -Dscale
If the store is set to raise exceptions, a lot of tests which expect rename(bad options) to return false now get exceptions. One of the contract tests would downgrade if the raised exception was FileAlreadyExistsException and the contract xml said that was ok. I'm reluctant to go with a bigger patch.
this PR is so that Hive and friends can get better reporting on errors, rather than have them lost. It will be optional
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r591325915
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestRenameDeleteRace.java
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonClientException;
+import org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
Review comment:
We need fix similar to #2758 after #2575.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r589956618
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md
##########
@@ -1126,6 +1126,26 @@ We also recommend using applications/application
options which do not rename files when committing work or when copying data
to S3, but instead write directly to the final destination.
+## Rename not behaving as "expected"
+
+S3 is not a filesystem. The S3A connector mimics rename by
+
+* HEAD then LIST of source path
+* HEAD then LIST of destination path
+* File-by-file copy of source objects to destination.
+ Parallelized, with page listings of directory objects and issuing of DELETE requests.
+* Post-delete recreation of destination parent directory marker, if needed.
Review comment:
recreation of **source** parent directory marker?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#issuecomment-795612632
ah, thanks -lovely. just tweaked the imports slightly for better backporting; I'll merge once the next compile is good, then do a cp and retest for branch-3.3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r591511493
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestRenameDeleteRace.java
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonClientException;
+import org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
Review comment:
no need, the compiler finds it. Will fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#issuecomment-794005013
> 3. don't downgrade FileNotFoundException to 'false' if source doesn't exist
> 4. raise FileAlreadyExistsException if dest path exists or parent is a file.
Should this JIRA be marked as incompatible for applicatinos assuming existing behavior?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#issuecomment-795576948
testing: s3 london, most recently with `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep -Ds3guard -Ddynamo -Dscale`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#issuecomment-793993687
LGTM overall. Tests against Tokyo region worked. I just left comments for nits.
```
$ mvn verify -Dtest=x -Dit.test=ITestS3AContractRename,ITestlRenameDeleteRace -Ds3guard
...
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestlRenameDeleteRace
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.044 s - in org.apache.hadoop.fs.s3a.impl.ITestlRenameDeleteRace
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.603 s - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[WARNING] Tests run: 22, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 12.523 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#issuecomment-793030939
Testing: s3 london with markers==keep and delete, s3guard on and off.
* change in error reporting of s3a FS matched with relevant changes in s3a.xml for contract tests.
* skip tests verifying that you can't rename 2+ levels under a file.
* Failures related to endpoints of common-crawl and ITestAssumeRole.testAssumeRoleBadInnerAuth: known and fixed in #2675.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r591334439
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestRenameDeleteRace.java
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonClientException;
+import org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
Review comment:
gotcha. Can we add some style checking to block this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r589957067
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestlRenameDeleteRace.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonClientException;
+import org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.AbstractS3ATestBase;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.util.BlockingThreadPoolExecutorService;
+
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY;
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY_DELETE;
+import static org.apache.hadoop.fs.s3a.Constants.S3GUARD_METASTORE_NULL;
+import static org.apache.hadoop.fs.s3a.Constants.S3_METADATA_STORE_IMPL;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.submit;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * HADOOP-16721: race condition with delete and rename underneath the same destination
+ * directory.
+ * This test suite recreates the failure using semaphores to guarantee the failure
+ * condition is encountered -then verifies that the rename operation is successful.
+ */
+public class ITestlRenameDeleteRace extends AbstractS3ATestBase {
Review comment:
Is the class name intended to be ITestRenameDeleteRace?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r590443239
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestlRenameDeleteRace.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonClientException;
+import org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.AbstractS3ATestBase;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.util.BlockingThreadPoolExecutorService;
+
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY;
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY_DELETE;
+import static org.apache.hadoop.fs.s3a.Constants.S3GUARD_METASTORE_NULL;
+import static org.apache.hadoop.fs.s3a.Constants.S3_METADATA_STORE_IMPL;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.submit;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * HADOOP-16721: race condition with delete and rename underneath the same destination
+ * directory.
+ * This test suite recreates the failure using semaphores to guarantee the failure
+ * condition is encountered -then verifies that the rename operation is successful.
+ */
+public class ITestlRenameDeleteRace extends AbstractS3ATestBase {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ITestlRenameDeleteRace.class);
+
+
+ /** Many threads for scale performance: {@value}. */
+ public static final int EXECUTOR_THREAD_COUNT = 2;
+
+ /**
+ * For submitting work.
+ */
+ private static final ListeningExecutorService EXECUTOR =
+ BlockingThreadPoolExecutorService.newInstance(
+ EXECUTOR_THREAD_COUNT,
+ EXECUTOR_THREAD_COUNT * 2,
+ 30, TimeUnit.SECONDS,
+ "test-operations");
+
+ @Override
+ protected Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+ String bucketName = getTestBucketName(conf);
+
+ removeBaseAndBucketOverrides(bucketName, conf,
+ S3_METADATA_STORE_IMPL,
+ DIRECTORY_MARKER_POLICY);
+ // use the keep policy to ensure that surplus markers exist
+ // to complicate failures
+ conf.set(DIRECTORY_MARKER_POLICY, DIRECTORY_MARKER_POLICY_DELETE);
+ conf.set(S3_METADATA_STORE_IMPL, S3GUARD_METASTORE_NULL);
+
+ return conf;
+ }
+
+ @Test
+ public void testDeleteRenameRaceCondition() throws Throwable {
+ describe("verify no race between delete and rename");
+ final S3AFileSystem fs = getFileSystem();
+ final Path path = path(getMethodName());
+ Path srcDir = new Path(path, "src");
+
+ // this dir must exist throughout the rename
+ Path destDir = new Path(path, "dest");
+ // this dir tree will be deleted in a thread which does not
+ // complete before the rename exists
+ Path destSubdir1 = new Path(destDir, "subdir1");
+ Path subfile1 = new Path(destSubdir1, "subfile1");
+
+ // this is the directory we want to copy over under the dest dir
+ Path srcSubdir2 = new Path(srcDir, "subdir2");
+ Path srcSubfile = new Path(srcSubdir2, "subfile2");
+ Path destSubdir2 = new Path(destDir, "subdir2");
+
+ // creates subfile1 and all parents
+ ContractTestUtils.touch(fs, subfile1);
+ assertIsDirectory(destDir);
+
+ // source subfile
+ ContractTestUtils.touch(fs, srcSubfile);
+
+ final BlockingFakeDirMarkerFS blockingFS
+ = new BlockingFakeDirMarkerFS();
+ blockingFS.initialize(fs.getUri(), fs.getConf());
+ // get the semaphore; this ensures that the next attempt to create
+ // a fake marker blocks
+ try {
+ blockingFS.blockBeforCreatingMarker.acquire();
+ final CompletableFuture<Path> future = submit(EXECUTOR, () -> {
+ LOG.info("deleting {}", destSubdir1);
+ blockingFS.delete(destSubdir1, true);
+ return destSubdir1;
+ });
+
+ // wait for the delete to complete the deletion phase
+ blockingFS.signalCreatingFakeParentDirectory.acquire();
+
+ // there is now no destination directory
+ assertPathDoesNotExist("should have been implicitly deleted", destDir);
+
+ try {
+ // Now attempt the rename in the normal FS.
+ LOG.info("renaming {} to {}", srcSubdir2, destSubdir2);
+ Assertions.assertThat(fs.rename(srcSubdir2, destSubdir2))
+ .describedAs("rename(%s, %s)", srcSubdir2, destSubdir2)
+ .isTrue();
+ } finally {
+ blockingFS.blockBeforCreatingMarker.release();
+ }
+
+ // now let the delete complete
+ LOG.info("Waiting for delete {} to finish", destSubdir1);
+ waitForCompletion(future);
+ assertPathExists("must now exist", destDir);
+ assertPathExists("must now exist", new Path(destSubdir2, "subfile2"));
+ assertPathDoesNotExist("Src dir deleted", srcSubdir2);
+
+ } finally {
+ cleanupWithLogger(LOG, blockingFS);
+ }
+
+ }
+
+ /**
+ * Subclass of S3A FS whose execution of maybeCreateFakeParentDirectory
+ * can be choreographed with another thread so as to reliably
+ * create the delete/rename race condition.
+ */
+ private final class BlockingFakeDirMarkerFS extends S3AFileSystem {
+
+ /**
+ * Block for entry into maybeCreateFakeParentDirectory(); will be released
+ * then.
+ */
+ private final Semaphore signalCreatingFakeParentDirectory = new Semaphore(
+ 1);
+
+ /**
+ * Semaphore to acquire before the marker can be listed/created.
+ */
+ private final Semaphore blockBeforCreatingMarker = new Semaphore(1);
Review comment:
done. My inability to type well surfaces again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#issuecomment-794003742
I got error on ITestS3AContractDistCp but this seems not to be related. I can not reproduce the error by running the ITestS3AContractDistCp alone by `-Dit.test=ITestS3AContractDistCp`.
```
$ mvn verify -Dtest=x -Dit.test=ITestS3AContract*
...
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 47.658 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[ERROR] testNonDirectWrite(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp) Time elapsed: 4.055 s <<< FAILURE!
java.lang.AssertionError: Expected 2 renames for a non-direct write distcp expected:<2> but was:<0>
at org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp.testNonDirectWrite(ITestS3AContractDistCp.java:98)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r590442351
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md
##########
@@ -1126,6 +1126,26 @@ We also recommend using applications/application
options which do not rename files when committing work or when copying data
to S3, but instead write directly to the final destination.
+## Rename not behaving as "expected"
+
+S3 is not a filesystem. The S3A connector mimics rename by
+
+* HEAD then LIST of source path
+* HEAD then LIST of destination path
+* File-by-file copy of source objects to destination.
+ Parallelized, with page listings of directory objects and issuing of DELETE requests.
+* Post-delete recreation of destination parent directory marker, if needed.
Review comment:
oops. fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org