You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Hafiz Mujadid <ha...@gmail.com> on 2015/07/13 06:41:25 UTC
Drill with S3 without hardcoding credentials into core-site
Hi Drill Community!
I am trying to use apache drill to query data of multiple users placed on
s3.
I successfully connected Drill to S3 by placing access and secret keys in
core-site.xml.
Is it possible to use Drill with S3 without hardcoding credentials into
core-site like defining credentials for multiple users on the fly?
Thanks
Re: Drill with S3 without hardcoding credentials into core-site
Posted by Hafiz Mujadid <ha...@gmail.com>.
Thanks Paul, I will try it and inform you about the progress.
thanks :)
On Tue, Jul 14, 2015 at 12:24 AM, Paul Mogren <PM...@commercehub.com>
wrote:
> In the storage plugin configuration that you have for S3, the connection
> URL can be like “s3n://accessKeyId:secretKeyId@bucket”.
> AFAIK, there is nothing stopping you from creating multiple such
> configurations pointing to the same bucket with different credentials.
>
> These can even be created dynamically with a REST invocation.
> It’s the “on the fly” part of your question that is your challenge, I
> think. Maybe you can get by without that, or maybe you can safely automate
> the above.
>
>
> On 7/13/15, 3:01 PM, "Hafiz Mujadid" <ha...@gmail.com> wrote:
>
> >paul can you guide me further what are u trying to say?
> >
> >
> >
> >On Mon, Jul 13, 2015 at 11:46 PM, Paul Mogren <PM...@commercehub.com>
> >wrote:
> >
> >> Your question seems to be about clients passing credentials to a Drill
> >> query which are then passed through to resources as they are accessed. I
> >> don¹t think you are going to find that. You can put the credentials in
> >>the
> >> URL in storage plugin configuration instead, if that helps. Storage
> >> plugins can be dynamically managed, at least.
> >>
> >>
> >>
> >>
> >>
> >> On 7/13/15, 2:19 AM, "Ted Dunning" <te...@gmail.com> wrote:
> >>
> >> >On Sun, Jul 12, 2015 at 9:41 PM, Hafiz Mujadid
> >><ha...@gmail.com>
> >> >wrote:
> >> >
> >> >> I successfully connected Drill to S3 by placing access and secret
> >>keys
> >> >>in
> >> >> core-site.xml.
> >> >>
> >> >> Is it possible to use Drill with S3 without hardcoding credentials
> >>into
> >> >> core-site like defining credentials for multiple users on the fly?
> >> >>
> >> >
> >> >Not sure if there is.
> >> >
> >> >But please do make sure that you issue IAM credentials for this purpose
> >> >that are highly limited in what they can do.
> >>
> >>
> >
> >
> >--
> >Regards: HAFIZ MUJADID
>
>
--
Regards: HAFIZ MUJADID
Re: Drill with S3 without hardcoding credentials into core-site
Posted by Paul Mogren <PM...@commercehub.com>.
Hafiz,
I have not worked with this at all, but wanted to make sure you saw
https://drill.apache.org/docs/configuring-user-impersonation/ and
https://drill.apache.org/docs/configuring-user-authentication/
Maybe there is an implementation of PAM that integrates with IAM...
-Paul
On 7/13/15, 3:24 PM, "Paul Mogren" <PM...@commercehub.com> wrote:
>In the storage plugin configuration that you have for S3, the connection
>URL can be like “s3n://accessKeyId:secretKeyId@bucket”.
>AFAIK, there is nothing stopping you from creating multiple such
>configurations pointing to the same bucket with different credentials.
>
>These can even be created dynamically with a REST invocation.
>It’s the “on the fly” part of your question that is your challenge, I
>think. Maybe you can get by without that, or maybe you can safely automate
>the above.
>
>
>On 7/13/15, 3:01 PM, "Hafiz Mujadid" <ha...@gmail.com> wrote:
>
>>paul can you guide me further what are u trying to say?
>>
>>
>>
>>On Mon, Jul 13, 2015 at 11:46 PM, Paul Mogren <PM...@commercehub.com>
>>wrote:
>>
>>> Your question seems to be about clients passing credentials to a Drill
>>> query which are then passed through to resources as they are accessed.
>>>I
>>> don¹t think you are going to find that. You can put the credentials in
>>>the
>>> URL in storage plugin configuration instead, if that helps. Storage
>>> plugins can be dynamically managed, at least.
>>>
>>>
>>>
>>>
>>>
>>> On 7/13/15, 2:19 AM, "Ted Dunning" <te...@gmail.com> wrote:
>>>
>>> >On Sun, Jul 12, 2015 at 9:41 PM, Hafiz Mujadid
>>><ha...@gmail.com>
>>> >wrote:
>>> >
>>> >> I successfully connected Drill to S3 by placing access and secret
>>>keys
>>> >>in
>>> >> core-site.xml.
>>> >>
>>> >> Is it possible to use Drill with S3 without hardcoding credentials
>>>into
>>> >> core-site like defining credentials for multiple users on the fly?
>>> >>
>>> >
>>> >Not sure if there is.
>>> >
>>> >But please do make sure that you issue IAM credentials for this
>>>purpose
>>> >that are highly limited in what they can do.
>>>
>>>
>>
>>
>>--
>>Regards: HAFIZ MUJADID
>
Re: Drill with S3 without hardcoding credentials into core-site
Posted by Paul Mogren <PM...@commercehub.com>.
In the storage plugin configuration that you have for S3, the connection
URL can be like “s3n://accessKeyId:secretKeyId@bucket”.
AFAIK, there is nothing stopping you from creating multiple such
configurations pointing to the same bucket with different credentials.
These can even be created dynamically with a REST invocation.
It’s the “on the fly” part of your question that is your challenge, I
think. Maybe you can get by without that, or maybe you can safely automate
the above.
On 7/13/15, 3:01 PM, "Hafiz Mujadid" <ha...@gmail.com> wrote:
>paul can you guide me further what are u trying to say?
>
>
>
>On Mon, Jul 13, 2015 at 11:46 PM, Paul Mogren <PM...@commercehub.com>
>wrote:
>
>> Your question seems to be about clients passing credentials to a Drill
>> query which are then passed through to resources as they are accessed. I
>> don¹t think you are going to find that. You can put the credentials in
>>the
>> URL in storage plugin configuration instead, if that helps. Storage
>> plugins can be dynamically managed, at least.
>>
>>
>>
>>
>>
>> On 7/13/15, 2:19 AM, "Ted Dunning" <te...@gmail.com> wrote:
>>
>> >On Sun, Jul 12, 2015 at 9:41 PM, Hafiz Mujadid
>><ha...@gmail.com>
>> >wrote:
>> >
>> >> I successfully connected Drill to S3 by placing access and secret
>>keys
>> >>in
>> >> core-site.xml.
>> >>
>> >> Is it possible to use Drill with S3 without hardcoding credentials
>>into
>> >> core-site like defining credentials for multiple users on the fly?
>> >>
>> >
>> >Not sure if there is.
>> >
>> >But please do make sure that you issue IAM credentials for this purpose
>> >that are highly limited in what they can do.
>>
>>
>
>
>--
>Regards: HAFIZ MUJADID
Re: Drill with S3 without hardcoding credentials into core-site
Posted by Hafiz Mujadid <ha...@gmail.com>.
paul can you guide me further what are u trying to say?
On Mon, Jul 13, 2015 at 11:46 PM, Paul Mogren <PM...@commercehub.com>
wrote:
> Your question seems to be about clients passing credentials to a Drill
> query which are then passed through to resources as they are accessed. I
> don¹t think you are going to find that. You can put the credentials in the
> URL in storage plugin configuration instead, if that helps. Storage
> plugins can be dynamically managed, at least.
>
>
>
>
>
> On 7/13/15, 2:19 AM, "Ted Dunning" <te...@gmail.com> wrote:
>
> >On Sun, Jul 12, 2015 at 9:41 PM, Hafiz Mujadid <ha...@gmail.com>
> >wrote:
> >
> >> I successfully connected Drill to S3 by placing access and secret keys
> >>in
> >> core-site.xml.
> >>
> >> Is it possible to use Drill with S3 without hardcoding credentials into
> >> core-site like defining credentials for multiple users on the fly?
> >>
> >
> >Not sure if there is.
> >
> >But please do make sure that you issue IAM credentials for this purpose
> >that are highly limited in what they can do.
>
>
--
Regards: HAFIZ MUJADID
Re: Drill with S3 without hardcoding credentials into core-site
Posted by Paul Mogren <PM...@commercehub.com>.
Your question seems to be about clients passing credentials to a Drill
query which are then passed through to resources as they are accessed. I
don¹t think you are going to find that. You can put the credentials in the
URL in storage plugin configuration instead, if that helps. Storage
plugins can be dynamically managed, at least.
On 7/13/15, 2:19 AM, "Ted Dunning" <te...@gmail.com> wrote:
>On Sun, Jul 12, 2015 at 9:41 PM, Hafiz Mujadid <ha...@gmail.com>
>wrote:
>
>> I successfully connected Drill to S3 by placing access and secret keys
>>in
>> core-site.xml.
>>
>> Is it possible to use Drill with S3 without hardcoding credentials into
>> core-site like defining credentials for multiple users on the fly?
>>
>
>Not sure if there is.
>
>But please do make sure that you issue IAM credentials for this purpose
>that are highly limited in what they can do.
Re: Drill with S3 without hardcoding credentials into core-site
Posted by Ted Dunning <te...@gmail.com>.
On Sun, Jul 12, 2015 at 9:41 PM, Hafiz Mujadid <ha...@gmail.com>
wrote:
> I successfully connected Drill to S3 by placing access and secret keys in
> core-site.xml.
>
> Is it possible to use Drill with S3 without hardcoding credentials into
> core-site like defining credentials for multiple users on the fly?
>
Not sure if there is.
But please do make sure that you issue IAM credentials for this purpose
that are highly limited in what they can do.