[jira] [Created] (IMPALA-13017) Add configuration option to set S3 endpoint for the minicluster

["Laszlo Gaal (Jira)" <ji...@apache.org>]

Laszlo Gaal created IMPALA-13017:
------------------------------------

             Summary: Add configuration option to set S3 endpoint for the minicluster
                 Key: IMPALA-13017
                 URL: https://issues.apache.org/jira/browse/IMPALA-13017
             Project: IMPALA
          Issue Type: Improvement
            Reporter: Laszlo Gaal
            Assignee: Laszlo Gaal


As AWS S3 adds functional improvements, some of the required configuration items become stricter, or optional items become mandatory. One example is the endpoint configuration: while older regions support the legacy endpoint syntax of _bucketname_.s3.amazonaws.com,  new regions, where S3 buckets can be created require the AWS v4 request signing algorithm, which requires the request URL to contain the region as well. In the latter case the bucket URL becomes _bucketname.region-code_.amazonaws.com, e.g. test-bucket.us-west-2.amazonaws.com.

Bucket configuration for the Hadoop Cloud Connector (for the s3a:// protocol) works a bit differently: the bucket name is supplied separately, e.g. s3a://landsat-pds, and various other parameters can be supplied via optional fields in core-site.xml

When the Impala minicluster is configured to use S3 for its default filesystem, it should be possible to supply the endpoint to use so that S3-specific tests can be run against endpoints that don't support connecting via the legacy generic endpoint.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org