[DISCUSS] REST: Refreshing vended credentials

[Eduard Tudenhöfner <et...@apache.org>]

Hey everyone,

I'd like to propose a mechanism and changes in order to be able to refresh
vended credentials for tables.

Please find the proposal doc here
<https://docs.google.com/document/d/1acCkaPCO7WsLtvYugrayurbef4zCnD2rb3ZPBKeaYoo/edit?usp=sharing>
.
The proposal requires a spec change, which can be seen in #11281
<https://github.com/apache/iceberg/pull/11281>.

As discussed in the last sync, this should hopefully help in better
understanding the proposal around standardizing credentials in the OpenAPI
spec, which is being discussed in
https://lists.apache.org/thread/jmklpnywnghg7qwmwr14zj2k6tnxmdo4.

Thanks,
Eduard

Re: [DISCUSS] REST: Refreshing vended credentials

[Eduard Tudenhöfner <et...@apache.org>]

Hey Yufei,

for 1) a client would choose the longest matching prefix. In terms of
failure I guess it really depends on what kind of credentials the server
sent to the client. If the server sent multiple credentials for the same
table (one generic (*prefix=s3*) and one narrowed-down one (
*prefix=s3://my-custom-bucket/...*)) and the client accidentally chose the
generic one, then it really depends on the underlying credential whether
the client fails or not.

For 2) I don't see any good reason to introduce any kind of versioning for
this. It's worth mentioning that we're planning to document
storage-specific configuration with #10576
<https://github.com/apache/iceberg/pull/10576>, so that should generally
solve this issue and servers/clients should know what to send/use.

Eduard

On Mon, Oct 14, 2024 at 7:34 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi Eduard,
>
> Thanks for the proposal. I'm excited about the new spec. I have two
> questions:
>
> 1. This is probably a dumb question due to the lack of context, but I'm a
> bit confused about how clients should select a prefix to use. In scenarios
> where multiple prefixes exist, which one should the clients choose? If a
> client selects the wrong prefix, what would be the implications?
> 2. The proposal suggests a completely schema-less approach for handling
> credentials using key-value pairs. How do we manage potential
> incompatibilities when the credential schema changes, such as adding a new
> key or renaming an existing one? The current design relies entirely on
> synchronization between the server and client implementations. While
> credential properties tend to be stable, changes can still occur. Ignoring
> this risk could result in client failures when server-side changes are
> made, especially since these credentials aren't versioned like API
> endpoints (e.g., v1, v2). Should we consider introducing versioning to
> mitigate this risk?
>
>
> Yufei
>
>
> On Thu, Oct 10, 2024 at 2:02 PM Jack Ye <ye...@gmail.com> wrote:
>
>> +1 for adding this in the REST spec.
>>
>> Glue has a similar API GetTemporaryGlueTableCredentials [1], which was
>> introduced because of performance and also security reasons. For example,
>> we don't want to propagate credentials across the compute nodes in the
>> cluster, and each compute node needs to fetch only the credentials
>> independently. Such an API becomes handy to do improvements like caching.
>>
>> Best,
>> Jack Ye
>>
>> [1]
>> https://docs.aws.amazon.com/cli/latest/reference/lakeformation/get-temporary-glue-table-credentials.html
>>
>>
>> On Thu, Oct 10, 2024 at 3:47 AM Eduard Tudenhöfner <
>> etudenhoefner@apache.org> wrote:
>>
>>> Hey everyone,
>>>
>>> I'd like to propose a mechanism and changes in order to be able to
>>> refresh vended credentials for tables.
>>>
>>> Please find the proposal doc here
>>> <https://docs.google.com/document/d/1acCkaPCO7WsLtvYugrayurbef4zCnD2rb3ZPBKeaYoo/edit?usp=sharing>
>>> .
>>> The proposal requires a spec change, which can be seen in #11281
>>> <https://github.com/apache/iceberg/pull/11281>.
>>>
>>> As discussed in the last sync, this should hopefully help in better
>>> understanding the proposal around standardizing credentials in the OpenAPI
>>> spec, which is being discussed in
>>> https://lists.apache.org/thread/jmklpnywnghg7qwmwr14zj2k6tnxmdo4.
>>>
>>> Thanks,
>>> Eduard
>>>
>>

Re: [DISCUSS] REST: Refreshing vended credentials

[Yufei Gu <fl...@gmail.com>]

Hi Eduard,

Thanks for the proposal. I'm excited about the new spec. I have two
questions:

1. This is probably a dumb question due to the lack of context, but I'm a
bit confused about how clients should select a prefix to use. In scenarios
where multiple prefixes exist, which one should the clients choose? If a
client selects the wrong prefix, what would be the implications?
2. The proposal suggests a completely schema-less approach for handling
credentials using key-value pairs. How do we manage potential
incompatibilities when the credential schema changes, such as adding a new
key or renaming an existing one? The current design relies entirely on
synchronization between the server and client implementations. While
credential properties tend to be stable, changes can still occur. Ignoring
this risk could result in client failures when server-side changes are
made, especially since these credentials aren't versioned like API
endpoints (e.g., v1, v2). Should we consider introducing versioning to
mitigate this risk?


Yufei


On Thu, Oct 10, 2024 at 2:02 PM Jack Ye <ye...@gmail.com> wrote:

> +1 for adding this in the REST spec.
>
> Glue has a similar API GetTemporaryGlueTableCredentials [1], which was
> introduced because of performance and also security reasons. For example,
> we don't want to propagate credentials across the compute nodes in the
> cluster, and each compute node needs to fetch only the credentials
> independently. Such an API becomes handy to do improvements like caching.
>
> Best,
> Jack Ye
>
> [1]
> https://docs.aws.amazon.com/cli/latest/reference/lakeformation/get-temporary-glue-table-credentials.html
>
>
> On Thu, Oct 10, 2024 at 3:47 AM Eduard Tudenhöfner <
> etudenhoefner@apache.org> wrote:
>
>> Hey everyone,
>>
>> I'd like to propose a mechanism and changes in order to be able to
>> refresh vended credentials for tables.
>>
>> Please find the proposal doc here
>> <https://docs.google.com/document/d/1acCkaPCO7WsLtvYugrayurbef4zCnD2rb3ZPBKeaYoo/edit?usp=sharing>
>> .
>> The proposal requires a spec change, which can be seen in #11281
>> <https://github.com/apache/iceberg/pull/11281>.
>>
>> As discussed in the last sync, this should hopefully help in better
>> understanding the proposal around standardizing credentials in the OpenAPI
>> spec, which is being discussed in
>> https://lists.apache.org/thread/jmklpnywnghg7qwmwr14zj2k6tnxmdo4.
>>
>> Thanks,
>> Eduard
>>
>

Re: [DISCUSS] REST: Refreshing vended credentials

[Jack Ye <ye...@gmail.com>]

+1 for adding this in the REST spec.

Glue has a similar API GetTemporaryGlueTableCredentials [1], which was
introduced because of performance and also security reasons. For example,
we don't want to propagate credentials across the compute nodes in the
cluster, and each compute node needs to fetch only the credentials
independently. Such an API becomes handy to do improvements like caching.

Best,
Jack Ye

[1]
https://docs.aws.amazon.com/cli/latest/reference/lakeformation/get-temporary-glue-table-credentials.html


On Thu, Oct 10, 2024 at 3:47 AM Eduard Tudenhöfner <et...@apache.org>
wrote:

> Hey everyone,
>
> I'd like to propose a mechanism and changes in order to be able to refresh
> vended credentials for tables.
>
> Please find the proposal doc here
> <https://docs.google.com/document/d/1acCkaPCO7WsLtvYugrayurbef4zCnD2rb3ZPBKeaYoo/edit?usp=sharing>
> .
> The proposal requires a spec change, which can be seen in #11281
> <https://github.com/apache/iceberg/pull/11281>.
>
> As discussed in the last sync, this should hopefully help in better
> understanding the proposal around standardizing credentials in the OpenAPI
> spec, which is being discussed in
> https://lists.apache.org/thread/jmklpnywnghg7qwmwr14zj2k6tnxmdo4.
>
> Thanks,
> Eduard
>