[jira] [Reopened] (HDFS-8727) Allow using path style addressing for accessing the s3 endpoint

["Stephen Montgomery (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephen Montgomery reopened HDFS-8727:
--------------------------------------
      Assignee:     (was: Andrew Baptist)

Hi,
I'd like to re-open this ticket please. I've done some further digging into this and believe that Andrew's original patch is still needed ie using a Hadoop S3A config property flag to "switch on" path style access in the underlying Amazon S3 client. Overriding the custom S3A endpoint has no effect (unless you specifically use an IPv4 address which is more a workaround/hack).

To force/trick the Amazon S3 client to use old path style access (instead of virtual hosting) you can use dodgy bucket names (eg '..', '.-' in the name, caps etc) and IPv4 addresses for the endpoint - see com.amazonaws.services.s3.AmazonS3Client. configRequest() method - pretty much making sure that the DNS lookups will fail for syntactic reasons.

I'm happy to update Andrew's original patch and supply a test case, if needed. Like Andrew mentioned, the test case will be of no real benefit as it will just exercising Amazon client functionality. It's also hard to do as the AWS client is pretty inaccessible around confirming the flag has been set.

Whats the process of re-opening this ticket? What Hadoop branch will this be targeted for ie it looks that 2.8 one has all of the S3A fixes...?

Thanks,
Stephen


> Allow using path style addressing for accessing the s3 endpoint
> ---------------------------------------------------------------
>
>                 Key: HDFS-8727
>                 URL: https://issues.apache.org/jira/browse/HDFS-8727
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.7.1
>            Reporter: Andrew Baptist
>              Labels: features
>         Attachments: hdfs-8728.patch.2
>
>
> There is no ability to specify using path style access for the s3 endpoint. There are numerous non-amazon implementations of storage that support the amazon API's but only support path style access such as Cleversafe and Ceph. Additionally in many environments it is difficult to configure DNS correctly to get virtual host style addressing to work



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)