S3 Storage Plugin not working for Signature V4 regions

[Ju...@continental-corporation.com]

Hello,

I am trying to configure the storage plugin for Amazon S3 to use it with 
Apache Drill version 1.7.0.
I would like to connect to an S3 bucket in eu-central-1 region which is 
only supporting signature V4.

I did lot's of tests and configurations and I cannot get it running. I 
always get the response "Bad request", "Error Code:400".

Even by setting the parameter

<property>
  <name>fs.s3a.endpoint</name>
  <value>s3.eu-central-1.amazonaws.com</value>
</property>

in the core-site.xml of Apache Drill install directory I get the "bad 
request" response.

I could successfully connect to buckets in different regions supporting V2 
and V4 signature, but I cannot connect to buckets in regions only 
supporting signature V4 (ap-northeast-2 and eu-central-1)

Thanks for your help,
Juergen

Re: S3 Storage Plugin not working for Signature V4 regions

[Uwe Korn <uw...@xhochy.com>]

Hello,

this could probably be that problem that the AWS SDK coming with the drill version is quite old. I’m doing some tests currently with Drill 1.8 + a custom Hadoop 3.0.0-alpha1 version that was patched to use the latest AWS SDK. That should not only give you access to V4 regions but it also improved the query performance of Drill+Parquet+S3 by magnitudes for me (that was simply down to some changes in the s3a:// implementation in Hadoop). 

Not sure if there is another way to easily upgrade the used AWS SDK without going the path of a custom Hadoop build but maybe someone else could help here.

Cheers
Uwe


> Am 29.09.2016 um 10:12 schrieb Juergen.Broda@continental-corporation.com:
> 
> Hello,
> 
> I am trying to configure the storage plugin for Amazon S3 to use it with 
> Apache Drill version 1.7.0.
> I would like to connect to an S3 bucket in eu-central-1 region which is 
> only supporting signature V4.
> 
> I did lot's of tests and configurations and I cannot get it running. I 
> always get the response "Bad request", "Error Code:400".
> 
> Even by setting the parameter
> 
> <property>
>  <name>fs.s3a.endpoint</name>
>  <value>s3.eu-central-1.amazonaws.com</value>
> </property>
> 
> in the core-site.xml of Apache Drill install directory I get the "bad 
> request" response.
> 
> I could successfully connect to buckets in different regions supporting V2 
> and V4 signature, but I cannot connect to buckets in regions only 
> supporting signature V4 (ap-northeast-2 and eu-central-1)
> 
> Thanks for your help,
> Juergen