You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by Алексей Рябов <ry...@gmail.com> on 2024/03/05 10:59:55 UTC

storage-based authentification

I am trying to understand the meaning of "storage-based"
authentication in kyuubi, which is, as documentation says
(https://kyuubi.readthedocs.io/en/v1.8.1-docs/security/authorization/spark/overview.html#storage-based-authorization),
enabled by default.
What i want to achieve is:
- user logs in to kyuubi
- kyuubi starts spark engine
- spark is configured to read/store data in apache iceberg format
- data is stored in s3 cluster (minio in my case)
- bucket policy is configured in minio to allow/deny access data
according to s3 credentials

Following is the spark configuration example, related to Iceberg:

    kyuubi.engineEnv.AWS_REGION=us-east-1
    kyuubi.engineEnv.AWS_ACCESS_KEY_ID=minioadmin
    kyuubi.engineEnv.AWS_SECRET_ACCESS_KEY=minioadmin
    spark.executorEnv.AWS_REGION=us-east-1
    spark.executorEnv.AWS_ACCESS_KEY_ID=minioadmin
    spark.executorEnv.AWS_SECRET_ACCESS_KEY=minioadmin

The question is: can I start spark using different s3 credentials for
each user? In my case i want to change values of
kyuubi.engineEnv.AWS_ACCESS_KEY_ID and
kyuubi.engineEnv.AWS_SECRET_ACCESS_KEY according to username and
password logged in to kyuubi.

Please advise.

Re: storage-based authentification

Posted by Алексей Рябов <ry...@gmail.com>.
Hi.


Yes, I have confused terms authorization and authentication...

Thank you for your reply, you helped a lot!  Option with implementing
custom conf advisor looks suitable for me, will take a closer look on it.

Thanks!

ср, 6 мар. 2024 г. в 06:18, Kent Yao <ya...@apache.org>:
>
> Hi,
> I guess you are looking at the doc in the wrong way.
> It is authorization, not authentication.
>
> Well, when Spark accesses s3, it does require
> authentication. Yes, we can send different creds to each engine.
> However, these configurations are not generated and passed to
> engines by default.
> The possible ways to do this are:
> - pass them manually by client users in connection configurations[1]
> - inject these by implementing a custom conf advisor [2]
>
> Thanks
>
> Kent Yao
> [1] https://kyuubi.readthedocs.io/en/master/configuration/settings.html#via-jdbc-connection-url
> [2] https://kyuubi.readthedocs.io/en/master/extensions/server/configuration.html
>
> On 2024/03/05 10:59:55 Алексей Рябов wrote:
> > I am trying to understand the meaning of "storage-based"
> > authentication in kyuubi, which is, as documentation says
> > (https://kyuubi.readthedocs.io/en/v1.8.1-docs/security/authorization/spark/overview.html#storage-based-authorization),
> > enabled by default.
> > What i want to achieve is:
> > - user logs in to kyuubi
> > - kyuubi starts spark engine
> > - spark is configured to read/store data in apache iceberg format
> > - data is stored in s3 cluster (minio in my case)
> > - bucket policy is configured in minio to allow/deny access data
> > according to s3 credentials
> >
> > Following is the spark configuration example, related to Iceberg:
> >
> >     kyuubi.engineEnv.AWS_REGION=us-east-1
> >     kyuubi.engineEnv.AWS_ACCESS_KEY_ID=minioadmin
> >     kyuubi.engineEnv.AWS_SECRET_ACCESS_KEY=minioadmin
> >     spark.executorEnv.AWS_REGION=us-east-1
> >     spark.executorEnv.AWS_ACCESS_KEY_ID=minioadmin
> >     spark.executorEnv.AWS_SECRET_ACCESS_KEY=minioadmin
> >
> > The question is: can I start spark using different s3 credentials for
> > each user? In my case i want to change values of
> > kyuubi.engineEnv.AWS_ACCESS_KEY_ID and
> > kyuubi.engineEnv.AWS_SECRET_ACCESS_KEY according to username and
> > password logged in to kyuubi.
> >
> > Please advise.
> >

Re: storage-based authentification

Posted by Kent Yao <ya...@apache.org>.
Hi,
I guess you are looking at the doc in the wrong way.
It is authorization, not authentication.

Well, when Spark accesses s3, it does require 
authentication. Yes, we can send different creds to each engine.
However, these configurations are not generated and passed to
engines by default.
The possible ways to do this are:
- pass them manually by client users in connection configurations[1]
- inject these by implementing a custom conf advisor [2]

Thanks

Kent Yao
[1] https://kyuubi.readthedocs.io/en/master/configuration/settings.html#via-jdbc-connection-url
[2] https://kyuubi.readthedocs.io/en/master/extensions/server/configuration.html

On 2024/03/05 10:59:55 Алексей Рябов wrote:
> I am trying to understand the meaning of "storage-based"
> authentication in kyuubi, which is, as documentation says
> (https://kyuubi.readthedocs.io/en/v1.8.1-docs/security/authorization/spark/overview.html#storage-based-authorization),
> enabled by default.
> What i want to achieve is:
> - user logs in to kyuubi
> - kyuubi starts spark engine
> - spark is configured to read/store data in apache iceberg format
> - data is stored in s3 cluster (minio in my case)
> - bucket policy is configured in minio to allow/deny access data
> according to s3 credentials
> 
> Following is the spark configuration example, related to Iceberg:
> 
>     kyuubi.engineEnv.AWS_REGION=us-east-1
>     kyuubi.engineEnv.AWS_ACCESS_KEY_ID=minioadmin
>     kyuubi.engineEnv.AWS_SECRET_ACCESS_KEY=minioadmin
>     spark.executorEnv.AWS_REGION=us-east-1
>     spark.executorEnv.AWS_ACCESS_KEY_ID=minioadmin
>     spark.executorEnv.AWS_SECRET_ACCESS_KEY=minioadmin
> 
> The question is: can I start spark using different s3 credentials for
> each user? In my case i want to change values of
> kyuubi.engineEnv.AWS_ACCESS_KEY_ID and
> kyuubi.engineEnv.AWS_SECRET_ACCESS_KEY according to username and
> password logged in to kyuubi.
> 
> Please advise.
>