You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by st...@apache.org on 2022/10/13 12:40:24 UTC
[hadoop] branch branch-3.3 updated: HADOOP-18292. Fix s3 select tests when running against unsupported storage class (#4489)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 52eca61a3e4 HADOOP-18292. Fix s3 select tests when running against unsupported storage class (#4489)
52eca61a3e4 is described below
commit 52eca61a3e472827d4b707aeeb0635ca4576195e
Author: monthonk <47...@users.noreply.github.com>
AuthorDate: Thu Oct 13 13:13:36 2022 +0100
HADOOP-18292. Fix s3 select tests when running against unsupported storage class (#4489)
Follow-on from HADOOP-12020.
Contributed by Monthon Klongklaew
---
.../src/site/markdown/tools/hadoop-aws/s3_select.md | 15 +++++++++++++++
.../apache/hadoop/fs/s3a/select/AbstractS3SelectTest.java | 10 ++++++++++
2 files changed, 25 insertions(+)
diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3_select.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3_select.md
index a5aaae91454..649c80a22d4 100644
--- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3_select.md
+++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3_select.md
@@ -934,6 +934,21 @@ Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: GZIP is not applic
...
```
+
+### AWSBadRequestException `UnsupportedStorageClass`
+
+S3 Select doesn't work with some storage classes like Glacier or Reduced Redundancy.
+Make sure you've set `fs.s3a.create.storage.class` to a supported storage class for S3 Select.
+
+```
+org.apache.hadoop.fs.s3a.AWSBadRequestException:
+ Select on s3a://example/dataset.csv.gz:
+ com.amazonaws.services.s3.model.AmazonS3Exception:
+ We do not support REDUCED_REDUNDANCY storage class.
+ Please check the service documentation and try again.
+ (Service: Amazon S3; Status Code: 400; Error Code: UnsupportedStorageClass
+```
+
### `PathIOException`: "seek() not supported"
The input stream returned by the select call does not support seeking
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/AbstractS3SelectTest.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/AbstractS3SelectTest.java
index bf5d96e73b3..2c1a10a21d0 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/AbstractS3SelectTest.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/AbstractS3SelectTest.java
@@ -60,7 +60,9 @@ import org.apache.hadoop.mapreduce.lib.input.LineRecordReader;
import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.DurationInfo;
+import static org.apache.hadoop.fs.s3a.Constants.STORAGE_CLASS;
import static org.apache.hadoop.fs.s3a.S3ATestUtils.getLandsatCSVPath;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
import static org.apache.hadoop.fs.s3a.select.CsvFile.ALL_QUOTES;
import static org.apache.hadoop.fs.s3a.select.SelectConstants.*;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
@@ -280,6 +282,14 @@ public abstract class AbstractS3SelectTest extends AbstractS3ATestBase {
.hasCapability(S3_SELECT_CAPABILITY);
}
+ @Override
+ protected Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+ removeBaseAndBucketOverrides(conf, STORAGE_CLASS);
+
+ return conf;
+ }
+
/**
* Setup: requires select to be available.
*/
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org