You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2022/07/15 10:21:00 UTC

[jira] [Created] (HADOOP-18339) S3A storage class option only picked up when buffering writes to disk

Steve Loughran created HADOOP-18339:
---------------------------------------

             Summary: S3A storage class option only picked up when buffering writes to disk
                 Key: HADOOP-18339
                 URL: https://issues.apache.org/jira/browse/HADOOP-18339
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs/s3
    Affects Versions: 3.3.9
            Reporter: Steve Loughran


when you switch s3a output stream buffering to heap or byte buffer, the storage class option isn't added to the put request


{code}

  <property>
    <name>fs.s3a.fast.upload.buffer</name>
    <value>bytebuffer</value>
  </property>

{code}

and the ITestS3AStorageClass tests fail.
{code}

java.lang.AssertionError: [Storage class of object s3a://stevel-london/test/testCreateAndCopyObjectWithStorageClassGlacier/file1] 
Expecting:
 <null>
to be equal to:
 <"glacier">
ignoring case considerations

	at org.apache.hadoop.fs.s3a.ITestS3AStorageClass.assertObjectHasStorageClass(ITestS3AStorageClass.java:215)
	at org.apache.hadoop.fs.s3a.ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier(ITestS3AStorageClass.java:129)


{code}

we noticed this in a code review; the request factory only sets the option when the source is a file, not memory.

proposed: parameterize the test suite on disk/byte buffer, then fix




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org