Drill connect to S3 with AWS EMR role

[Alonzo Barnett <ab...@lexmark.com>]

I am looking into using Drill with AWS EMR.  My organization uses IAM roles
with EMR in order to rotate security credentials.

In a first round of testing I successfully connected Drill to S3 on a
cluster without IAM role based credential rotation which required placing
access and secret keys in core-site.xml.

It is possible to still use Drill with S3 without hardcoding credentials
into core-site?  With emrfs there is some work by AWS to rotate
credentials, and I would like to exploit the modification to core-site if
possible.

-- 
v/r,
Alonzo Barnett

Re: Drill connect to S3 with AWS EMR role

[David Tucker <dt...@maprtech.com>]

The current version of Hadoop in EMR (both Apache and MapR) does not support the IAM authentication to S3 without the credentials in core-site.   I believe the support has been integrated into Hadoop 2.6 … so when the EMR distributions upgrade to that level, the access you request should be supported.

Did you successfully configure the drill-bit to use the full EMRFS jars, or did you default to the older jets3t support ?   If you have the classpath settings for full emrfs support, please share them with the group (and I will integrate that support into the MapR EMR bootstrap action under development for Drill).

Regards, 
   David

On May 11, 2015, at 6:55 PM, Alonzo Barnett <ab...@lexmark.com> wrote:

> I am looking into using Drill with AWS EMR.  My organization uses IAM roles
> with EMR in order to rotate security credentials.
> 
> In a first round of testing I successfully connected Drill to S3 on a
> cluster without IAM role based credential rotation which required placing
> access and secret keys in core-site.xml.
> 
> It is possible to still use Drill with S3 without hardcoding credentials
> into core-site?  With emrfs there is some work by AWS to rotate
> credentials, and I would like to exploit the modification to core-site if
> possible.
> 
> -- 
> v/r,
> Alonzo Barnett