Standalone metastore

[Sandhya Agarwal <wr...@gmail.com>]

Hello,

We want to leverage standalone metastore for our project, for which I want
to enable access to multiple S3 buckets, each with its own access key and
secret key. I am trying to access the metastore operations from a Java
client using HiveMetastoreClient and using the thrift metastore URI to
connect to the metastore service. For the database location and table
location, I am using the S3A file system scheme. I am trying to set the
access key and secret key for the S3 bucket location in the
HiveMetastoreClient, but the client side settings are never honoured and I
cannot figure out a way to do this. One way is to provide the per-bucket
access key and secret key on the server side in metastore-site.xml.
However, I want this to be dynamic as I want the metastore to be multi
tenant enabled and based on the current tenant user, I wish to provide
these keys through my client. I tried looking through the metastore source
code, but did not find a way to do this. Can this be done ?

I am using apache-hive-metastore-3.0.0 version.

Thank you,

Sandhya

Re: Standalone metastore

[Sandhya Agarwal <wr...@gmail.com>]

Thank you for the revert.

We do have a use case to provide tenant / bucket specific keys for a tenant
user session, without having to restart the MetaStore server when a new
tenant is provisioned.

Nevertheless, let me explore a bit around Sentry / Ranger too.

Thank you,
Sandhya

On Thu, Jul 12, 2018 at 3:51 AM Vihang Karajgaonkar <vi...@cloudera.com>
wrote:

> AFAIK currently, s3 keys cannot be session specific currently. They are
> loaded by the metastore server when it starts and it cannot be modified
> without a metastore server restart. In order to do this we will have make
> some code changes. I had created HIVE-16913 for this long time back but
> never got around working on it later (I can take a relook at it if there
> are use-cases in practice which would need this).
>
> The other workaround would be to have one master key configured at the
> server level and restrict user access to urls using Sentry or Ranger.
>
>
> On Wed, Jul 11, 2018 at 2:44 AM, Sandhya Agarwal <wr...@gmail.com>
> wrote:
>
>> Hello,
>>
>> We want to leverage standalone metastore for our project, for which I
>> want to enable access to multiple S3 buckets, each with its own access key
>> and secret key. I am trying to access the metastore operations from a Java
>> client using HiveMetastoreClient and using the thrift metastore URI to
>> connect to the metastore service. For the database location and table
>> location, I am using the S3A file system scheme. I am trying to set the
>> access key and secret key for the S3 bucket location in the
>> HiveMetastoreClient, but the client side settings are never honoured and I
>> cannot figure out a way to do this. One way is to provide the per-bucket
>> access key and secret key on the server side in metastore-site.xml.
>> However, I want this to be dynamic as I want the metastore to be multi
>> tenant enabled and based on the current tenant user, I wish to provide
>> these keys through my client. I tried looking through the metastore source
>> code, but did not find a way to do this. Can this be done ?
>>
>> I am using apache-hive-metastore-3.0.0 version.
>>
>> Thank you,
>>
>> Sandhya
>>
>>
>

Re: Standalone metastore

[Vihang Karajgaonkar <vi...@cloudera.com>]

AFAIK currently, s3 keys cannot be session specific currently. They are
loaded by the metastore server when it starts and it cannot be modified
without a metastore server restart. In order to do this we will have make
some code changes. I had created HIVE-16913 for this long time back but
never got around working on it later (I can take a relook at it if there
are use-cases in practice which would need this).

The other workaround would be to have one master key configured at the
server level and restrict user access to urls using Sentry or Ranger.


On Wed, Jul 11, 2018 at 2:44 AM, Sandhya Agarwal <wr...@gmail.com>
wrote:

> Hello,
>
> We want to leverage standalone metastore for our project, for which I want
> to enable access to multiple S3 buckets, each with its own access key and
> secret key. I am trying to access the metastore operations from a Java
> client using HiveMetastoreClient and using the thrift metastore URI to
> connect to the metastore service. For the database location and table
> location, I am using the S3A file system scheme. I am trying to set the
> access key and secret key for the S3 bucket location in the
> HiveMetastoreClient, but the client side settings are never honoured and I
> cannot figure out a way to do this. One way is to provide the per-bucket
> access key and secret key on the server side in metastore-site.xml.
> However, I want this to be dynamic as I want the metastore to be multi
> tenant enabled and based on the current tenant user, I wish to provide
> these keys through my client. I tried looking through the metastore source
> code, but did not find a way to do this. Can this be done ?
>
> I am using apache-hive-metastore-3.0.0 version.
>
> Thank you,
>
> Sandhya
>
>