[jira] [Created] (FLINK-10817) Upgrade presto dependency to support path-style access

["Adam Lamar (JIRA)" <ji...@apache.org>]

Adam Lamar created FLINK-10817:
----------------------------------

             Summary: Upgrade presto dependency to support path-style access
                 Key: FLINK-10817
                 URL: https://issues.apache.org/jira/browse/FLINK-10817
             Project: Flink
          Issue Type: Improvement
            Reporter: Adam Lamar


In order to use any given non-AWS s3 implementation backed by the presto s3 filesystem, it is necessary to set at least one configuration parameter in flink-conf.yaml:
 * presto.s3.endpoint: https://example.com

This appears to work as expected for hosted s3 alternatives.

In order to use a bring-your-own, self-hosted s3 alternative like [minio|https://www.minio.io/], at least two configuration parameters are required:
 * presto.s3.endpoint: https://example.com
 * presto.s3.path-style-access: true

However, the second path-style-access parameter doesn't work because the 0.185 version of presto doesn't support passing through that configuration option to the hive s3 client.

To work around the issue, path-style-access can be forced on the s3 client by using an IP address for the endpoint (instead of a hostname). Without this workaround, flink attempts to use the virtualhost-style at bucketname.example.com, which fails unless the expected DNS records exist.

To solve this problem and enable non-IP endpoints, upgrade the [pom|https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-presto/pom.xml#L36] to at least 0.186 which includes [this commit|[https://github.com/prestodb/presto/commit/0707f2f21a96d2fd30953fb3fa9a9a03f03d88bd.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)