You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Nitin S <ns...@gmail.com> on 2018/09/17 21:20:43 UTC
S3 Storage Plugin - IAM roles support
Hello,
I am trying to setup drill in distributed mode and access some parquet
files in S3. The instructions here:
https://drill.apache.org/docs/s3-storage-plugin/ do not provide any info
on what to do if we are using IAM roles for S3 (instead of static keys)
Kindly advise how to set that up.
Thanks,
Nitin
Re: S3 Storage Plugin - IAM roles support
Posted by Nitin S <ns...@gmail.com>.
After spending a day or two, I found the solution to the issue. We need to
add the following snippet in the core-site.xml
<property>
<name>fs.s3a.aws.credentials.provider</name>
<value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value>
</property>
This explicitly tells drill to use that mode of fetching credentials
instead of relying on env variables or explicit access keys.
I would recommend adding it to the official document so that other folks
can benefit from this. If there is a way to do it myself, let me know (I
can raise a pull request etc too)
Thanks,
Nitin
On Mon, Sep 17, 2018 at 5:24 PM Nitin S <ns...@gmail.com> wrote:
> Hi ,
>
> I did try that before and it did not work. I was not able to find it in
> the documentation
>
> Thanks
> Nitin
>
> On Mon, Sep 17, 2018 at 3:37 PM Khurram Faraaz <kf...@mapr.com> wrote:
>
>> Hi Nitin,
>>
>> You can read the details from here -
>> http://drill-user.incubator.apache.narkive.com/mP5GE7fq/s3-using-iam-roles
>>
>> A drill user hit the same issue in the past and he has documented the
>> steps
>> for others to benefit. Hope that helps.
>>
>> Regards,
>> Khurram
>>
>> On Mon, Sep 17, 2018 at 2:20 PM, Nitin S <ns...@gmail.com> wrote:
>>
>> > Hello,
>> >
>> > I am trying to setup drill in distributed mode and access some parquet
>> > files in S3. The instructions here:
>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__drill.
>> > apache.org_docs_s3-2Dstorage-2Dplugin_&d=DwIBaQ&c=
>> > cskdkSMqhcnjZxdQVpwTXg&r=H5JEl9vb-mBIjic10QAbDD2vkUUKAxjO6wZO322RtdI&m=
>> > I3BfnYvwCJajF5EHuHVLb6Pzlb1C_56Em6a6kgxv1Ac&s=
>> > hQMUN2MEoN9AGwiPvEjbZR7mnh37WG9x2XWhPPm3ky8&e= do not provide any info
>> > on what to do if we are using IAM roles for S3 (instead of static keys)
>> >
>> > Kindly advise how to set that up.
>> >
>> > Thanks,
>> > Nitin
>> >
>>
>
Re: S3 Storage Plugin - IAM roles support
Posted by Nitin S <ns...@gmail.com>.
Hi ,
I did try that before and it did not work. I was not able to find it in the
documentation
Thanks
Nitin
On Mon, Sep 17, 2018 at 3:37 PM Khurram Faraaz <kf...@mapr.com> wrote:
> Hi Nitin,
>
> You can read the details from here -
> http://drill-user.incubator.apache.narkive.com/mP5GE7fq/s3-using-iam-roles
>
> A drill user hit the same issue in the past and he has documented the steps
> for others to benefit. Hope that helps.
>
> Regards,
> Khurram
>
> On Mon, Sep 17, 2018 at 2:20 PM, Nitin S <ns...@gmail.com> wrote:
>
> > Hello,
> >
> > I am trying to setup drill in distributed mode and access some parquet
> > files in S3. The instructions here:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__drill.
> > apache.org_docs_s3-2Dstorage-2Dplugin_&d=DwIBaQ&c=
> > cskdkSMqhcnjZxdQVpwTXg&r=H5JEl9vb-mBIjic10QAbDD2vkUUKAxjO6wZO322RtdI&m=
> > I3BfnYvwCJajF5EHuHVLb6Pzlb1C_56Em6a6kgxv1Ac&s=
> > hQMUN2MEoN9AGwiPvEjbZR7mnh37WG9x2XWhPPm3ky8&e= do not provide any info
> > on what to do if we are using IAM roles for S3 (instead of static keys)
> >
> > Kindly advise how to set that up.
> >
> > Thanks,
> > Nitin
> >
>
Re: S3 Storage Plugin - IAM roles support
Posted by Khurram Faraaz <kf...@mapr.com>.
Hi Nitin,
You can read the details from here -
http://drill-user.incubator.apache.narkive.com/mP5GE7fq/s3-using-iam-roles
A drill user hit the same issue in the past and he has documented the steps
for others to benefit. Hope that helps.
Regards,
Khurram
On Mon, Sep 17, 2018 at 2:20 PM, Nitin S <ns...@gmail.com> wrote:
> Hello,
>
> I am trying to setup drill in distributed mode and access some parquet
> files in S3. The instructions here:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__drill.
> apache.org_docs_s3-2Dstorage-2Dplugin_&d=DwIBaQ&c=
> cskdkSMqhcnjZxdQVpwTXg&r=H5JEl9vb-mBIjic10QAbDD2vkUUKAxjO6wZO322RtdI&m=
> I3BfnYvwCJajF5EHuHVLb6Pzlb1C_56Em6a6kgxv1Ac&s=
> hQMUN2MEoN9AGwiPvEjbZR7mnh37WG9x2XWhPPm3ky8&e= do not provide any info
> on what to do if we are using IAM roles for S3 (instead of static keys)
>
> Kindly advise how to set that up.
>
> Thanks,
> Nitin
>