You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by st...@apache.org on 2021/04/24 17:28:53 UTC
[hadoop] branch branch-3.3 updated (389d303 -> 1b2bc77)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a change to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 389d303 HADOOP-17471. ABFS to collect IOStatistics (#2731) (#2950)
new 33f9ceb HADOOP-17136. ITestS3ADirectoryPerformance.testListOperations failing (#2153)
new 1b2bc77 HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java | 2 +-
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md | 4 ++--
.../src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java | 1 +
.../org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java | 2 +-
4 files changed, 5 insertions(+), 4 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 02/02: HADOOP-17454. [s3a] Disable bucket existence check
- set fs.s3a.bucket.probe to 0 (#2593)
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 1b2bc77923b176c7fa84946c21df486dd3859a06
Author: Gabor Bota <ga...@cloudera.com>
AuthorDate: Tue Jan 5 15:43:01 2021 +0100
HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593)
Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, because the test would fail by default otherwise.
Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
---
.../hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java | 2 +-
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md | 4 ++--
.../src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java | 1 +
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
index 67e5c17..c4b8f6e 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
@@ -518,7 +518,7 @@ public final class Constants {
* will be validated using {@code S3AFileSystem.verifyBucketExistsV2()}.
* Value: {@value}
*/
- public static final int S3A_BUCKET_PROBE_DEFAULT = 2;
+ public static final int S3A_BUCKET_PROBE_DEFAULT = 0;
/**
* How long a directory listing in the MS is considered as authoritative.
diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
index 2813235..9258c47 100644
--- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
+++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
@@ -1040,9 +1040,9 @@ options are covered in [Testing](./testing.md).
<property>
<name>fs.s3a.bucket.probe</name>
- <value>2</value>
+ <value>0</value>
<description>
- The value can be 0, 1 or 2 (default).
+ The value can be 0 (default), 1 or 2.
When set to 0, bucket existence checks won't be done
during initialization thus making it faster.
Though it should be noted that when the bucket is not available in S3,
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
index 57cbbcb..eb68eed 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
@@ -159,6 +159,7 @@ public class ITestS3AConfiguration {
return intercept(clazz,
() -> {
fs = S3ATestUtils.createTestFileSystem(conf);
+ fs.listFiles(new Path("/"), false);
return "expected failure creating FS " + text + " got " + fs;
});
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/02: HADOOP-17136.
ITestS3ADirectoryPerformance.testListOperations failing (#2153)
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 33f9ceb3cc81e63327226fe87583af464e0ed7b0
Author: Mukund Thakur <mt...@cloudera.com>
AuthorDate: Mon Jul 20 21:28:50 2020 +0530
HADOOP-17136. ITestS3ADirectoryPerformance.testListOperations failing (#2153)
A regression caused by HADOOP-17022: the reduction in LIST calls broken an assertion.
Contributed by Mukund Thakur
Change-Id: Ib9725165906931634567fd1f62a81e3a6ea5620c
---
.../org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java
index 0751959..61f6ef3 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java
@@ -141,7 +141,7 @@ public class ITestS3ADirectoryPerformance extends S3AScaleTestBase {
listStatusCalls,
getFileStatusCalls);
if (!fs.hasMetadataStore()) {
- assertEquals(listRequests.toString(), 2, listRequests.diff());
+ assertEquals(listRequests.toString(), 1, listRequests.diff());
}
reset(metadataRequests,
listRequests,
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org