You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by st...@apache.org on 2025/04/03 14:18:36 UTC

(hadoop) branch branch-3.4 updated (fc5b23336b7 -> f3ec55b5911)

This is an automated email from the ASF dual-hosted git repository.

stevel pushed a change to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git


    from fc5b23336b7 HADOOP-19444. [ABFS][FnsOverBlob][Tests] Add Tests For Negative Scenarios Identified for Ingress Operations  (#7572)
     new 1ce05fd0961 HADOOP-19485. S3A: Upgrade AWS V2 SDK to 2.29.52 (#7479)
     new 7646d889aae HADOOP-19485. S3A: Test Failures after SDK 2.29.52 Upgrade
     new 02925dd4efe HADOOP-19455. S3A: Enable logging of SDK client metrics
     new f3ec55b5911 HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 LICENSE-binary                                     |  3 +-
 hadoop-project/pom.xml                             |  2 +-
 .../hadoop/fs/s3a/DefaultS3ClientFactory.java      | 11 +++-
 .../hadoop/fs/s3a/impl/AwsSdkWorkarounds.java      | 16 +++--
 .../hadoop/fs/s3a/impl/UploadContentProviders.java |  4 +-
 .../tools/hadoop-aws/third_party_stores.md         | 15 ++++-
 .../tools/hadoop-aws/troubleshooting_s3a.md        | 41 +++++++++++++
 .../hadoop/fs/s3a/ITestS3AConfiguration.java       |  2 +
 .../hadoop/fs/s3a/ITestS3AEncryptionSSEC.java      |  2 +
 .../hadoop/fs/s3a/ITestS3AEndpointRegion.java      | 69 ++++++++++++++++------
 .../org/apache/hadoop/fs/s3a/S3ATestUtils.java     |  2 +-
 .../hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java | 37 ++----------
 .../hadoop/fs/s3a/tools/ITestBucketTool.java       |  1 +
 .../org/apache/hadoop/fs/sdk/TestAWSV2SDK.java     |  2 +-
 .../hadoop-aws/src/test/resources/log4j.properties |  4 ++
 15 files changed, 146 insertions(+), 65 deletions(-)


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


(hadoop) 03/04: HADOOP-19455. S3A: Enable logging of SDK client metrics

Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 02925dd4efe53b38cbdc736d082299b11df06624
Author: Steve Loughran <st...@cloudera.com>
AuthorDate: Fri Mar 7 18:41:50 2025 +0000

    HADOOP-19455. S3A: Enable logging of SDK client metrics
    
    To log the output of the AWS SDK metrics, set the log
    `org.apache.hadoop.fs.s3a.DefaultS3ClientFactory` to `TRACE`.
    
    Contributed by Steve Loughran.
---
 .../hadoop/fs/s3a/DefaultS3ClientFactory.java      | 11 +++++-
 .../tools/hadoop-aws/troubleshooting_s3a.md        | 41 ++++++++++++++++++++++
 .../hadoop-aws/src/test/resources/log4j.properties |  4 +++
 3 files changed, 55 insertions(+), 1 deletion(-)

diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
index 1acb2b9945e..f52093abbe7 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
@@ -38,6 +38,7 @@
 import software.amazon.awssdk.http.auth.spi.scheme.AuthScheme;
 import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
 import software.amazon.awssdk.identity.spi.AwsCredentialsIdentity;
+import software.amazon.awssdk.metrics.LoggingMetricPublisher;
 import software.amazon.awssdk.regions.Region;
 import software.amazon.awssdk.services.s3.S3AsyncClient;
 import software.amazon.awssdk.services.s3.S3AsyncClientBuilder;
@@ -201,12 +202,20 @@ private <BuilderT extends S3BaseClientBuilder<BuilderT, ClientT>, ClientT> Build
     final ClientOverrideConfiguration.Builder override =
         createClientOverrideConfiguration(parameters, conf);
 
-    S3BaseClientBuilder s3BaseClientBuilder = builder
+    S3BaseClientBuilder<BuilderT, ClientT> s3BaseClientBuilder = builder
         .overrideConfiguration(override.build())
         .credentialsProvider(parameters.getCredentialSet())
         .disableS3ExpressSessionAuth(!parameters.isExpressCreateSession())
         .serviceConfiguration(serviceConfiguration);
 
+    if (LOG.isTraceEnabled()) {
+      // if this log is set to "trace" then we turn on logging of SDK metrics.
+      // The metrics itself will log at info; it is just that reflection work
+      // would be needed to change that setting safely for shaded and unshaded aws artifacts.
+      s3BaseClientBuilder.overrideConfiguration(o ->
+          o.addMetricPublisher(LoggingMetricPublisher.create()));
+    }
+
     if (conf.getBoolean(HTTP_SIGNER_ENABLED, HTTP_SIGNER_ENABLED_DEFAULT)) {
       // use an http signer through an AuthScheme
       final AuthScheme<AwsCredentialsIdentity> signer =
diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md
index 55ebf7614c1..1bdb2c66cb4 100644
--- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md
+++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md
@@ -1359,6 +1359,47 @@ execchain.MainClientExec (MainClientExec.java:execute(284)) - Connection can be
 
 ```
 
+To log the output of the AWS SDK metrics, set the log
+`org.apache.hadoop.fs.s3a.DefaultS3ClientFactory` to `TRACE`.
+This will then turn on logging of the internal SDK metrics.
+
+These will actually be logged at INFO in the log
+```
+software.amazon.awssdk.metrics.LoggingMetricPublisher
+```
+
+```text
+INFO  metrics.LoggingMetricPublisher (LoggerAdapter.java:info(165)) - Metrics published:
+MetricCollection(name=ApiCall, metrics=[
+MetricRecord(metric=MarshallingDuration, value=PT0.000092041S),
+MetricRecord(metric=RetryCount, value=0),
+MetricRecord(metric=ApiCallSuccessful, value=true),
+MetricRecord(metric=OperationName, value=DeleteObject),
+MetricRecord(metric=EndpointResolveDuration, value=PT0.000132792S),
+MetricRecord(metric=ApiCallDuration, value=PT0.064890875S),
+MetricRecord(metric=CredentialsFetchDuration, value=PT0.000017458S),
+MetricRecord(metric=ServiceEndpoint, value=https://buckets3.eu-west-2.amazonaws.com),
+MetricRecord(metric=ServiceId, value=S3)], children=[
+MetricCollection(name=ApiCallAttempt, metrics=[
+    MetricRecord(metric=TimeToFirstByte, value=PT0.06260225S),
+    MetricRecord(metric=SigningDuration, value=PT0.000293083S),
+    MetricRecord(metric=ReadThroughput, value=0.0),
+    MetricRecord(metric=ServiceCallDuration, value=PT0.06260225S),
+    MetricRecord(metric=HttpStatusCode, value=204),
+    MetricRecord(metric=BackoffDelayDuration, value=PT0S),
+    MetricRecord(metric=TimeToLastByte, value=PT0.064313667S),
+    MetricRecord(metric=AwsRequestId, value=RKZD44SE5DW91K1G)], children=[
+        MetricCollection(name=HttpClient, metrics=[
+        MetricRecord(metric=AvailableConcurrency, value=1),
+        MetricRecord(metric=LeasedConcurrency, value=0),
+        MetricRecord(metric=ConcurrencyAcquireDuration, value=PT0S),
+        MetricRecord(metric=PendingConcurrencyAcquires, value=0),
+        MetricRecord(metric=MaxConcurrency, value=512),
+        MetricRecord(metric=HttpClientName, value=Apache)], children=[])
+    ])
+  ])
+```
+
 ### <a name="audit-logging"></a> Enable S3 Server-side Logging
 
 The [Auditing](auditing) feature of the S3A connector can be used to generate
diff --git a/hadoop-tools/hadoop-aws/src/test/resources/log4j.properties b/hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
index 7b8dd3c11fc..f61668643a1 100644
--- a/hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
+++ b/hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
@@ -102,3 +102,7 @@ log4j.logger.org.apache.hadoop.fs.s3a.S3AStorageStatistics=INFO
 # services it launches itself.
 # log4.logger.org.apache.hadoop.service=DEBUG
 
+# log this at trace to trigger enabling printing of the low-level
+# performance metrics in the AWS SDK itself.
+# log4j.logger.org.apache.hadoop.fs.s3a.DefaultS3ClientFactory=TRACE
+


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


(hadoop) 02/04: HADOOP-19485. S3A: Test Failures after SDK 2.29.52 Upgrade

Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 7646d889aaeb55620e1f5267fb5cd8f849968bf6
Author: Steve Loughran <st...@cloudera.com>
AuthorDate: Fri Mar 7 17:21:40 2025 +0000

    HADOOP-19485. S3A: Test Failures after SDK 2.29.52 Upgrade
    
    Code changes related to HADOOP-19485.
    
    AwsSdkWorkarounds no longer needs to cut back on transfer manager logging
    (HADOOP-19272) :
    - Remove log downgrade and change assertion to expect nothing to be logged.
    - remove false positives from log.
    
    ITestS3AEndpointRegion failure:
    - Change in state of AwsExecutionAttribute.ENDPOINT_OVERRIDDEN
      attribute requires test tuning to match.
    
    Some tests against third-party stores fail
    - Includes fix for the assumeStoreAwsHosted() logic.
    - Documents how to turn off multipart uploads with third-party stores.
    
    Contributed by Steve Loughran.
---
 .../hadoop/fs/s3a/impl/AwsSdkWorkarounds.java      | 16 +++--
 .../tools/hadoop-aws/third_party_stores.md         | 15 ++++-
 .../hadoop/fs/s3a/ITestS3AConfiguration.java       |  2 +
 .../hadoop/fs/s3a/ITestS3AEncryptionSSEC.java      |  2 +
 .../hadoop/fs/s3a/ITestS3AEndpointRegion.java      | 69 ++++++++++++++++------
 .../org/apache/hadoop/fs/s3a/S3ATestUtils.java     |  2 +-
 .../hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java | 37 ++----------
 .../hadoop/fs/s3a/tools/ITestBucketTool.java       |  1 +
 .../org/apache/hadoop/fs/sdk/TestAWSV2SDK.java     |  2 +-
 9 files changed, 86 insertions(+), 60 deletions(-)

diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AwsSdkWorkarounds.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AwsSdkWorkarounds.java
index a0673b123b2..fabb543bd90 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AwsSdkWorkarounds.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AwsSdkWorkarounds.java
@@ -18,9 +18,10 @@
 
 package org.apache.hadoop.fs.s3a.impl;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 import org.apache.hadoop.classification.VisibleForTesting;
-import org.apache.hadoop.fs.s3a.impl.logging.LogControl;
-import org.apache.hadoop.fs.s3a.impl.logging.LogControllerFactory;
 
 /**
  * This class exists to support workarounds for parts of the AWS SDK
@@ -35,16 +36,20 @@ public final class AwsSdkWorkarounds {
   public static final String TRANSFER_MANAGER =
       "software.amazon.awssdk.transfer.s3.S3TransferManager";
 
+  private static final Logger LOG = LoggerFactory.getLogger(AwsSdkWorkarounds.class);
+
   private AwsSdkWorkarounds() {
   }
 
   /**
    * Prepare logging before creating AWS clients.
+   * There is currently no logging to require tuning,
+   * so this only logs at trace that it was invoked.
    * @return true if the log tuning operation took place.
    */
   public static boolean prepareLogging() {
-    return LogControllerFactory.createController().
-        setLogLevel(TRANSFER_MANAGER, LogControl.LogLevel.ERROR);
+    LOG.trace("prepareLogging()");
+    return true;
   }
 
   /**
@@ -53,7 +58,6 @@ public static boolean prepareLogging() {
    */
   @VisibleForTesting
   static boolean restoreNoisyLogging() {
-    return LogControllerFactory.createController().
-        setLogLevel(TRANSFER_MANAGER, LogControl.LogLevel.INFO);
+    return true;
   }
 }
diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/third_party_stores.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/third_party_stores.md
index 1018ec9e7d6..feffdf0a8b0 100644
--- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/third_party_stores.md
+++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/third_party_stores.md
@@ -40,6 +40,7 @@ The features which may be unavailable include:
   This is now the default -do not change it.
 * List API to use (`fs.s3a.list.version = 1`)
 * Bucket lifecycle rules to clean up pending uploads.
+* Support for multipart uploads.
 
 ### Disabling Change Detection
 
@@ -409,7 +410,7 @@ which is a subset of the AWS API.
 To get a compatible access and secret key, follow the instructions of
 [Simple migration from Amazon S3 to Cloud Storage](https://cloud.google.com/storage/docs/aws-simple-migration#defaultproj).
 
-Here are the per-bucket setings for an example bucket "gcs-container"
+Here are the per-bucket settings for an example bucket "gcs-container"
 in Google Cloud Storage. Note the multiobject delete option must be disabled;
 this makes renaming and deleting significantly slower.
 
@@ -452,11 +453,21 @@ this makes renaming and deleting significantly slower.
     <value>true</value>
   </property>
 
+  <!-- any value is allowed here, using "gcs" is more informative -->
   <property>
     <name>fs.s3a.bucket.gcs-container.endpoint.region</name>
-    <value>dummy</value>
+    <value>gcs</value>
   </property>
 
+  <!-- multipart uploads trigger 400 response-->
+  <property>
+    <name>fs.s3a.multipart.uploads.enabled</name>
+    <value>false</value>
+  </property>
+    <property>
+    <name>fs.s3a.optimized.copy.from.local.enabled</name>
+    <value>false</value>
+  </property>
 </configuration>
 ```
 
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
index 97af80e70a5..e6dabc91bac 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
@@ -589,6 +589,8 @@ public void testS3SpecificSignerOverride() throws Exception {
     config.set(AWS_REGION, EU_WEST_1);
     disableFilesystemCaching(config);
     fs = S3ATestUtils.createTestFileSystem(config);
+    assumeStoreAwsHosted(fs);
+
 
     S3Client s3Client = getS3Client("testS3SpecificSignerOverride");
 
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java
index 8671d962175..0f79881466f 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java
@@ -39,6 +39,7 @@
 import static org.apache.hadoop.fs.s3a.Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM;
 import static org.apache.hadoop.fs.s3a.Constants.SERVER_SIDE_ENCRYPTION_KEY;
 
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.assumeStoreAwsHosted;
 import static org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName;
 import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeSkipRootTests;
 import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
@@ -101,6 +102,7 @@ public void setup() throws Exception {
     // although not a root dir test, this confuses paths enough it shouldn't be run in
     // parallel with other jobs
     maybeSkipRootTests(getConfiguration());
+    assumeStoreAwsHosted(getFileSystem());
   }
 
   @Override
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
index 07caeb02f41..62f2ffbc0df 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
@@ -24,9 +24,9 @@
 import java.nio.file.AccessDeniedException;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
 
 import org.assertj.core.api.Assertions;
-import org.junit.Ignore;
 import org.junit.Test;
 import software.amazon.awssdk.awscore.AwsExecutionAttribute;
 import software.amazon.awssdk.awscore.exception.AwsServiceException;
@@ -55,6 +55,7 @@
 import static org.apache.hadoop.fs.s3a.Constants.S3_ENCRYPTION_ALGORITHM;
 import static org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.ERROR_ENDPOINT_WITH_FIPS;
 import static org.apache.hadoop.fs.s3a.S3ATestUtils.assume;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.assumeStoreAwsHosted;
 import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
 import static org.apache.hadoop.fs.s3a.test.PublicDatasetTestUtils.DEFAULT_REQUESTER_PAYS_BUCKET_NAME;
 import static org.apache.hadoop.io.IOUtils.closeStream;
@@ -106,6 +107,10 @@ public class ITestS3AEndpointRegion extends AbstractS3ATestBase {
 
   public static final String EXCEPTION_THROWN_BY_INTERCEPTOR = "Exception thrown by interceptor";
 
+  /**
+   * Text to include in assertions.
+   */
+  private static final AtomicReference<String> EXPECTED_MESSAGE = new AtomicReference<>();
   /**
    * New FS instance which will be closed in teardown.
    */
@@ -477,6 +482,7 @@ public void testCentralEndpointAndNullRegionWithCRUD() throws Throwable {
     describe("Access the test bucket using central endpoint and"
         + " null region, perform file system CRUD operations");
     final Configuration conf = getConfiguration();
+    assumeStoreAwsHosted(getFileSystem());
 
     final Configuration newConf = new Configuration(conf);
 
@@ -499,6 +505,7 @@ public void testCentralEndpointAndNullRegionWithCRUD() throws Throwable {
   public void testCentralEndpointAndNullRegionFipsWithCRUD() throws Throwable {
     describe("Access the test bucket using central endpoint and"
         + " null region and fips enabled, perform file system CRUD operations");
+    assumeStoreAwsHosted(getFileSystem());
 
     final String bucketLocation = getFileSystem().getBucketLocation();
     assume("FIPS can be enabled to access buckets from US or Canada endpoints only",
@@ -576,7 +583,7 @@ private void assertOpsUsingNewFs() throws IOException {
         .isFalse();
   }
 
-  private final class RegionInterceptor implements ExecutionInterceptor {
+  private static final class RegionInterceptor implements ExecutionInterceptor {
     private final String endpoint;
     private final String region;
     private final boolean isFips;
@@ -591,28 +598,49 @@ private final class RegionInterceptor implements ExecutionInterceptor {
     public void beforeExecution(Context.BeforeExecution context,
         ExecutionAttributes executionAttributes)  {
 
-      if (endpoint != null && !endpoint.endsWith(CENTRAL_ENDPOINT)) {
-        Assertions.assertThat(
-                executionAttributes.getAttribute(AwsExecutionAttribute.ENDPOINT_OVERRIDDEN))
-            .describedAs("Endpoint not overridden").isTrue();
 
-        Assertions.assertThat(
-                executionAttributes.getAttribute(AwsExecutionAttribute.CLIENT_ENDPOINT).toString())
-            .describedAs("There is an endpoint mismatch").isEqualTo("https://" + endpoint);
+      // extract state from the execution attributes.
+      final Boolean endpointOveridden =
+          executionAttributes.getAttribute(AwsExecutionAttribute.ENDPOINT_OVERRIDDEN);
+      final String clientEndpoint =
+          executionAttributes.getAttribute(AwsExecutionAttribute.CLIENT_ENDPOINT).toString();
+      final Boolean fipsEnabled = executionAttributes.getAttribute(
+          AwsExecutionAttribute.FIPS_ENDPOINT_ENABLED);
+      final String reg = executionAttributes.getAttribute(AwsExecutionAttribute.AWS_REGION).
+          toString();
+
+      String state = "SDK beforeExecution callback; "
+          + "endpointOveridden=" + endpointOveridden
+          + "; clientEndpoint=" + clientEndpoint
+          + "; fipsEnabled=" + fipsEnabled
+          + "; region=" + reg;
+
+      if (endpoint != null && !endpoint.endsWith(CENTRAL_ENDPOINT)) {
+        Assertions.assertThat(endpointOveridden)
+            .describedAs("Endpoint not overridden in %s. Client Config=%s",
+                state, EXPECTED_MESSAGE.get())
+            .isTrue();
+
+        Assertions.assertThat(clientEndpoint)
+            .describedAs("There is an endpoint mismatch in %s. Client Config=%s",
+                state, EXPECTED_MESSAGE.get())
+            .isEqualTo("https://" + endpoint);
       } else {
-        Assertions.assertThat(
-                executionAttributes.getAttribute(AwsExecutionAttribute.ENDPOINT_OVERRIDDEN))
-            .describedAs("Endpoint is overridden").isEqualTo(null);
+        Assertions.assertThat(endpointOveridden)
+            .describedAs("Attribute endpointOveridden is null in %s. Client Config=%s",
+                state, EXPECTED_MESSAGE.get())
+            .isEqualTo(false);
       }
 
-      Assertions.assertThat(
-              executionAttributes.getAttribute(AwsExecutionAttribute.AWS_REGION).toString())
-          .describedAs("Incorrect region set").isEqualTo(region);
+      Assertions.assertThat(reg)
+          .describedAs("Incorrect region set in %s. Client Config=%s",
+              state, EXPECTED_MESSAGE.get())
+          .isEqualTo(region);
 
       // verify the fips state matches expectation.
-      Assertions.assertThat(executionAttributes.getAttribute(
-          AwsExecutionAttribute.FIPS_ENDPOINT_ENABLED))
-          .describedAs("Incorrect FIPS flag set in execution attributes")
+      Assertions.assertThat(fipsEnabled)
+          .describedAs("Incorrect FIPS flag set in %s; Client Config=%s",
+              state, EXPECTED_MESSAGE.get())
           .isNotNull()
           .isEqualTo(isFips);
 
@@ -637,6 +665,11 @@ private S3Client createS3Client(Configuration conf,
       String endpoint, String configuredRegion, String expectedRegion, boolean isFips)
       throws IOException {
 
+    String expected =
+        "endpoint=" + endpoint + "; region=" + configuredRegion
+        + "; expectedRegion=" + expectedRegion + "; isFips=" + isFips;
+    LOG.info("Creating S3 client with {}", expected);
+    EXPECTED_MESSAGE.set(expected);
     List<ExecutionInterceptor> interceptors = new ArrayList<>();
     interceptors.add(new RegionInterceptor(endpoint, expectedRegion, isFips));
 
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
index 6570b3ee056..1be33d1ff8e 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
@@ -1170,7 +1170,7 @@ public static void assumeNotS3ExpressFileSystem(final FileSystem fs) {
    */
   public static void assumeStoreAwsHosted(final FileSystem fs) {
     assume("store is not AWS S3",
-        !NetworkBinding.isAwsEndpoint(fs.getConf()
+        NetworkBinding.isAwsEndpoint(fs.getConf()
             .getTrimmed(ENDPOINT, DEFAULT_ENDPOINT)));
   }
 
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java
index ed7a32928b8..d18a722a0e2 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java
@@ -32,12 +32,9 @@
 import static org.apache.hadoop.test.GenericTestUtils.LogCapturer.captureLogs;
 
 /**
- * Verify that noisy transfer manager logs are turned off.
+ * Tests for any AWS SDK workaround code.
  * <p>
- * This is done by creating new FS instances and then
- * requesting an on-demand transfer manager from the store.
- * As this is only done once per FS instance, a new FS is
- * required per test case.
+ * These tests are inevitably brittle against SDK updates.
  */
 public class ITestAwsSdkWorkarounds extends AbstractS3ATestBase {
 
@@ -53,13 +50,6 @@ public class ITestAwsSdkWorkarounds extends AbstractS3ATestBase {
   private static final Logger XFER_LOG =
       LoggerFactory.getLogger(AwsSdkWorkarounds.TRANSFER_MANAGER);
 
-  /**
-   * This is the string which keeps being printed.
-   * {@value}.
-   */
-  private static final String FORBIDDEN =
-      "The provided S3AsyncClient is an instance of MultipartS3AsyncClient";
-
   /**
    * Marginal test run speedup by skipping needless test dir cleanup.
    * @throws IOException failure
@@ -70,23 +60,7 @@ protected void deleteTestDirInTeardown() throws IOException {
   }
 
   /**
-   * Test instantiation with logging disabled.
-   */
-  @Test
-  public void testQuietLogging() throws Throwable {
-    // simulate the base state of logging
-    noisyLogging();
-    // creating a new FS switches to quiet logging
-    try (S3AFileSystem newFs = newFileSystem()) {
-      String output = createAndLogTransferManager(newFs);
-      Assertions.assertThat(output)
-          .describedAs("LOG output")
-          .doesNotContain(FORBIDDEN);
-    }
-  }
-
-  /**
-   * Test instantiation with logging disabled.
+   * Test instantiation with logging enabled.
    */
   @Test
   public void testNoisyLogging() throws Throwable {
@@ -95,9 +69,8 @@ public void testNoisyLogging() throws Throwable {
       noisyLogging();
       String output = createAndLogTransferManager(newFs);
       Assertions.assertThat(output)
-          .describedAs("LOG output does not contain the forbidden text."
-              + " Has the SDK been fixed?")
-          .contains(FORBIDDEN);
+          .describedAs("LOG output")
+          .isEmpty();
     }
   }
 
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/tools/ITestBucketTool.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/tools/ITestBucketTool.java
index 50ffce7d87a..b37e6eec7c8 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/tools/ITestBucketTool.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/tools/ITestBucketTool.java
@@ -157,6 +157,7 @@ public void testSimpleBucketWithZoneParam() throws Throwable {
 
   @Test
   public void testS3ExpressBucketWithoutZoneParam() throws Throwable {
+    assumeStoreAwsHosted(getFileSystem());
     expectErrorCode(EXIT_USAGE,
         intercept(ExitUtil.ExitException.class, NO_ZONE_SUPPLIED, () ->
             bucketTool.exec("bucket", d(CREATE),
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java
index fca9fcc300c..1ddd3377cf0 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java
@@ -57,7 +57,7 @@ public void testShadedClasses() throws IOException {
     assertThat(v2ClassPath)
             .as("AWS V2 SDK should be present on the classpath").isNotNull();
     List<String> listOfV2SdkClasses = getClassNamesFromJarFile(v2ClassPath);
-    String awsSdkPrefix = "software/amazon/awssdk";
+    String awsSdkPrefix = "software/amazon/";
     List<String> unshadedClasses = new ArrayList<>();
     for (String awsSdkClass : listOfV2SdkClasses) {
       if (!awsSdkClass.startsWith(awsSdkPrefix)) {


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


(hadoop) 04/04: HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.

Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit f3ec55b5911d1e5e7da438c64f614a59a7db62a3
Author: Steve Loughran <st...@cloudera.com>
AuthorDate: Wed Mar 26 17:37:22 2025 +0000

    HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
    
    Downgrade warning message from INFO to DEBUG.
    
    Note: this does not fix the apparent issue, merely stops the log message
    being printed during single-object PUTs to S3 Express storage.
    
    Contributed by Steve Loughran.
---
 .../java/org/apache/hadoop/fs/s3a/impl/UploadContentProviders.java    | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/UploadContentProviders.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/UploadContentProviders.java
index d1fb28257f2..9716a09158e 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/UploadContentProviders.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/UploadContentProviders.java
@@ -286,7 +286,9 @@ public final InputStream newStream() {
         // the stream has been recreated for the first time.
         // notify only once for this stream, so as not to flood
         // the logs.
-        LOG.info("Stream recreated: {}", this);
+        // originally logged at info; logs at debug because HADOOP-19516
+        // means that this message is very common with S3 Express stores.
+        LOG.debug("Stream recreated: {}", this);
       }
       return setCurrentStream(createNewStream());
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


(hadoop) 01/04: HADOOP-19485. S3A: Upgrade AWS V2 SDK to 2.29.52 (#7479)

Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 1ce05fd09617d9851028575f622805f0e9eb6352
Author: Steve Loughran <st...@cloudera.com>
AuthorDate: Fri Mar 7 10:53:03 2025 +0000

    HADOOP-19485. S3A: Upgrade AWS V2 SDK to 2.29.52 (#7479)
    
    This upgrades to the final release the AWS SDK which
    retains compatibility with third-party stores.
    
    Note: this patch MUST be followed by
    * HADOOP-19485. S3A: Test failures after SDK upgrade
    
    And SHOULD be followed up with
    * HADOOP-19455. S3A: SDK client to log metrics at TRACE
    * HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
    
    Contributed by Steve Loughran.
---
 LICENSE-binary         | 3 +--
 hadoop-project/pom.xml | 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/LICENSE-binary b/LICENSE-binary
index 317698b43f3..66908996eda 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -361,8 +361,7 @@ org.objenesis:objenesis:2.6
 org.xerial.snappy:snappy-java:1.1.10.4
 org.yaml:snakeyaml:2.0
 org.wildfly.openssl:wildfly-openssl:2.1.4.Final
-software.amazon.awssdk:bundle:2.25.53
-
+software.amazon.awssdk:bundle:2.29.52
 
 --------------------------------------------------------------------------------
 This product bundles various third-party components under other open source
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index b848d26653d..b07c91e7868 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -204,7 +204,7 @@
     <make-maven-plugin.version>1.0-beta-1</make-maven-plugin.version>
     <surefire.fork.timeout>900</surefire.fork.timeout>
     <aws-java-sdk.version>1.12.720</aws-java-sdk.version>
-    <aws-java-sdk-v2.version>2.25.53</aws-java-sdk-v2.version>
+    <aws-java-sdk-v2.version>2.29.52</aws-java-sdk-v2.version>
     <amazon-s3-encryption-client-java.version>3.1.1</amazon-s3-encryption-client-java.version>
     <amazon-s3-analyticsaccelerator-s3.version>0.0.4</amazon-s3-analyticsaccelerator-s3.version>
     <aws.eventstream.version>1.0.1</aws.eventstream.version>


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org