You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Christian Thiel <ch...@hansetag.com.INVALID> on 2024/09/18 08:11:51 UTC
[DISCUSS] REST: OAuth2 Authentication Guide
Dear everyone,
the Iceberg REST specification allows for different ways of Authentication, OAuth2 is one of them. Until recently the OAuth2 /token endpoint was part of the REST-spec together with datatypes required for the client-credential flow. Both have since been removed from the spec for security reasons [2].
Probably because it was a part of the spec before, clients today typically only implement the client-credential flow. This stays behind OAuth2 capabilities and is unsuitable for human users. Common IdPs do not implement the client-credential flow for principals of human users for good reasons.
To mitigate this problem, we propose an extension of the Iceberg documentation in 3 steps. This proposal is neither an extension of the Iceberg REST Catalog specification nor OAuth2 itself. The Iceberg REST specification already specifies OAuth2 Authentication [3], which includes all the flows mentioned in the document of this proposal [1].
My proposal to go forward is as follows:
1. Use this proposals Google Doc for alignment in the community: [1]
https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
Discuss in a catalog sync in 1-2 weeks.
2. Condense consensus found in Google Doc to .md and add it to docs
3. Implement additional flows in the iceberg-(java, python, rust ..) packages.
For Java there is already a PR that goes in this direction which could use some more attention: https://github.com/apache/iceberg/pull/10753
For other languages I am not aware of any initiatives.
4. Encourage clients to allow configuration of new flows for users
Any feedback welcome!
Thanks
- Christian
[1]: https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
[2]: https://docs.google.com/document/d/1Xi5MRk8WdBWFC3N_eSmVcrLhk3yu5nJ9x_wC0ec6kVQ
[3]: https://github.com/apache/iceberg/blob/ed73ec43dd25c9023069ea1d3381a6d9229be53a/open-api/rest-catalog-open-api.yaml#L61
Re: [DISCUSS] REST: OAuth2 Authentication Guide
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Christian
Nice document, thanks !
Definitely a great idea to "document" the OAuth2 flow. My only comment
is that we should document the client side (what you are doing great
in the doc), but also the server side (it might help to understand the
full picture).
I propose to have a group effort on the document with you.
Thanks again !
Regards
JB
On Wed, Sep 18, 2024 at 10:11 AM Christian Thiel
<ch...@hansetag.com.invalid> wrote:
>
> Dear everyone,
>
>
> the Iceberg REST specification allows for different ways of Authentication, OAuth2 is one of them. Until recently the OAuth2 /token endpoint was part of the REST-spec together with datatypes required for the client-credential flow. Both have since been removed from the spec for security reasons [2].
>
> Probably because it was a part of the spec before, clients today typically only implement the client-credential flow. This stays behind OAuth2 capabilities and is unsuitable for human users. Common IdPs do not implement the client-credential flow for principals of human users for good reasons.
>
>
>
> To mitigate this problem, we propose an extension of the Iceberg documentation in 3 steps. This proposal is neither an extension of the Iceberg REST Catalog specification nor OAuth2 itself. The Iceberg REST specification already specifies OAuth2 Authentication [3], which includes all the flows mentioned in the document of this proposal [1].
>
>
>
> My proposal to go forward is as follows:
>
> Use this proposals Google Doc for alignment in the community: [1]
> https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
> Discuss in a catalog sync in 1-2 weeks.
> Condense consensus found in Google Doc to .md and add it to docs
> Implement additional flows in the iceberg-(java, python, rust ..) packages.
> For Java there is already a PR that goes in this direction which could use some more attention: https://github.com/apache/iceberg/pull/10753
> For other languages I am not aware of any initiatives.
> Encourage clients to allow configuration of new flows for users
>
> Any feedback welcome!
>
> Thanks
> - Christian
>
> [1]: https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
> [2]: https://docs.google.com/document/d/1Xi5MRk8WdBWFC3N_eSmVcrLhk3yu5nJ9x_wC0ec6kVQ
>
> [3]: https://github.com/apache/iceberg/blob/ed73ec43dd25c9023069ea1d3381a6d9229be53a/open-api/rest-catalog-open-api.yaml#L61
>
>
Re: [DISCUSS] REST: OAuth2 Authentication Guide
Posted by Christian Thiel <ch...@hansetag.com.INVALID>.
Thank you for your Feedback everyone!
It would be great if we could get some more eyes from the community on the server-side token exchange section at the bottom of the Document. Are we aware of any OAuth2 secured implementations that provide tokens from the resource server to the client apart from tabular?
https://docs.google.com/document/d/1buW9PCNoHPeP7Br5_vZRTU-_3TExwLx6bs075gi94xc/edit?usp=sharing
@Dimitri I agree that I wasn’t quite clear enough that catalogs IdPs might choose to not implement all of OAuth2. As you suggested I added a hint intended for users at the top of the document.
Re: [DISCUSS] REST: OAuth2 Authentication Guide
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
Hi Christian,
Thanks for pushing this initiative forward. I think it is quite useful.
I added some rather minor comments to the doc.
One bigger aspect of this, I guess, is that the doc currently talks about
what clients should do. This is important, of course. However, if a client
is able to obtain an access token via OAuth2 flows, it does not
automatically mean that any catalog implementation will be able to accept
such a token. What do you think about adding a section to clarify that
servers are free to support integrations with IdPs of their choosing but it
is not guaranteed and that users should check the documentation of the
catalog implementation about what exactly is supported with respect to
OAuth2.
WDYT?
Thanks,
Dmitri.
On Fri, Oct 25, 2024 at 12:25 PM Christian Thiel
<ch...@hansetag.com.invalid> wrote:
> Thanks everyone for your Feedback in the Catalog Sync and afterwards!
>
> I tried to address most of the Feedback and updated the Document.
>
>
>
> - The updated Document can be found here [1]:
>
> https://docs.google.com/document/d/1buW9PCNoHPeP7Br5_vZRTU-_3TExwLx6bs075gi94xc/edit?usp=sharing
> - It is linked now to an improvement (#11286 [2]) which also contains
> much of the Motivation which does not need to be part of the docs itself.
>
>
>
> I would ask all interested parties to leave comments in the google doc. If
> further clarification is needed, we can also discuss it again in the
> catalog sync.
>
>
>
> [1]
> https://docs.google.com/document/d/1buW9PCNoHPeP7Br5_vZRTU-_3TExwLx6bs075gi94xc/edit?usp=sharing
>
> [2] https://github.com/apache/iceberg/issues/11286
>
>
>
>
>
>
>
> *From: *Yufei Gu <fl...@gmail.com>
> *Date: *Saturday, 12. October 2024 at 12:30
> *To: *dev@iceberg.apache.org <de...@iceberg.apache.org>
> *Subject: *Re: [DISCUSS] REST: OAuth2 Authentication Guide
>
> Thanks Christian. Nice write-up! Authentication is essential to a
> production env. It's great to document it well given a lot of people don't
> necessarily have enough OAthen2 knowledge. Looking forward to the doc PRs
> and other client side changes.
>
>
> Yufei
>
>
>
>
>
> On Wed, Sep 18, 2024 at 8:31 AM Dmitri Bourlatchkov
> <dm...@dremio.com.invalid> wrote:
>
> Hi Christian,
>
>
>
> Very nice proposal. Thanks for putting it together! I added some comments
> to the doc.
>
>
>
> I think it is related to PR #10753 [4], which proposes some foundational
> refactoring to the java REST client to enable further enhancements in
> OAuth2 flows.
>
>
>
> Cheers,
>
> Dmitri.
>
>
>
> [4] https://github.com/apache/iceberg/pull/10753
>
>
>
> On Wed, Sep 18, 2024 at 4:12 AM Christian Thiel
> <ch...@hansetag.com.invalid> wrote:
>
> Dear everyone,
>
>
> the Iceberg REST specification allows for different ways of
> Authentication, OAuth2 is one of them. Until recently the OAuth2 /token
> endpoint was part of the REST-spec together with datatypes required for the
> client-credential flow. Both have since been removed from the spec for
> security reasons [2].
>
> Probably because it was a part of the spec before, clients today typically
> only implement the client-credential flow. This stays behind OAuth2
> capabilities and is unsuitable for human users. Common IdPs do not
> implement the client-credential flow for principals of human users for good
> reasons.
>
>
>
> To mitigate this problem, we propose an extension of the Iceberg
> documentation in 3 steps. This proposal is neither an extension of the
> Iceberg REST Catalog specification nor OAuth2 itself. The Iceberg REST
> specification already specifies OAuth2 Authentication [3], which includes
> all the flows mentioned in the document of this proposal [1].
>
>
>
> My proposal to go forward is as follows:
>
> 1. Use this proposals Google Doc for alignment in the community: [1]
>
> https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
> Discuss in a catalog sync in 1-2 weeks.
> 2. Condense consensus found in Google Doc to .md and add it to docs
> 3. Implement additional flows in the iceberg-(java, python, rust ..)
> packages.
> For Java there is already a PR that goes in this direction which could
> use some more attention: https://github.com/apache/iceberg/pull/10753
> For other languages I am not aware of any initiatives.
> 4. Encourage clients to allow configuration of new flows for users
>
> Any feedback welcome!
>
> Thanks
> - Christian
>
> [1]:
> https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
> [2]:
> https://docs.google.com/document/d/1Xi5MRk8WdBWFC3N_eSmVcrLhk3yu5nJ9x_wC0ec6kVQ
>
> [3]:
> https://github.com/apache/iceberg/blob/ed73ec43dd25c9023069ea1d3381a6d9229be53a/open-api/rest-catalog-open-api.yaml#L61
>
>
>
>
Re: [DISCUSS] REST: OAuth2 Authentication Guide
Posted by Christian Thiel <ch...@hansetag.com.INVALID>.
Thanks everyone for your Feedback in the Catalog Sync and afterwards!
I tried to address most of the Feedback and updated the Document.
* The updated Document can be found here [1]:
https://docs.google.com/document/d/1buW9PCNoHPeP7Br5_vZRTU-_3TExwLx6bs075gi94xc/edit?usp=sharing
* It is linked now to an improvement (#11286 [2]) which also contains much of the Motivation which does not need to be part of the docs itself.
I would ask all interested parties to leave comments in the google doc. If further clarification is needed, we can also discuss it again in the catalog sync.
[1] https://docs.google.com/document/d/1buW9PCNoHPeP7Br5_vZRTU-_3TExwLx6bs075gi94xc/edit?usp=sharing
[2] https://github.com/apache/iceberg/issues/11286
From: Yufei Gu <fl...@gmail.com>
Date: Saturday, 12. October 2024 at 12:30
To: dev@iceberg.apache.org <de...@iceberg.apache.org>
Subject: Re: [DISCUSS] REST: OAuth2 Authentication Guide
Thanks Christian. Nice write-up! Authentication is essential to a production env. It's great to document it well given a lot of people don't necessarily have enough OAthen2 knowledge. Looking forward to the doc PRs and other client side changes.
Yufei
On Wed, Sep 18, 2024 at 8:31 AM Dmitri Bourlatchkov <dm...@dremio.com.invalid> wrote:
Hi Christian,
Very nice proposal. Thanks for putting it together! I added some comments to the doc.
I think it is related to PR #10753 [4], which proposes some foundational refactoring to the java REST client to enable further enhancements in OAuth2 flows.
Cheers,
Dmitri.
[4] https://github.com/apache/iceberg/pull/10753
On Wed, Sep 18, 2024 at 4:12 AM Christian Thiel <ch...@hansetag.com.invalid> wrote:
Dear everyone,
the Iceberg REST specification allows for different ways of Authentication, OAuth2 is one of them. Until recently the OAuth2 /token endpoint was part of the REST-spec together with datatypes required for the client-credential flow. Both have since been removed from the spec for security reasons [2].
Probably because it was a part of the spec before, clients today typically only implement the client-credential flow. This stays behind OAuth2 capabilities and is unsuitable for human users. Common IdPs do not implement the client-credential flow for principals of human users for good reasons.
To mitigate this problem, we propose an extension of the Iceberg documentation in 3 steps. This proposal is neither an extension of the Iceberg REST Catalog specification nor OAuth2 itself. The Iceberg REST specification already specifies OAuth2 Authentication [3], which includes all the flows mentioned in the document of this proposal [1].
My proposal to go forward is as follows:
1. Use this proposals Google Doc for alignment in the community: [1]
https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
Discuss in a catalog sync in 1-2 weeks.
2. Condense consensus found in Google Doc to .md and add it to docs
3. Implement additional flows in the iceberg-(java, python, rust ..) packages.
For Java there is already a PR that goes in this direction which could use some more attention: https://github.com/apache/iceberg/pull/10753
For other languages I am not aware of any initiatives.
4. Encourage clients to allow configuration of new flows for users
Any feedback welcome!
Thanks
- Christian
[1]: https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
[2]: https://docs.google.com/document/d/1Xi5MRk8WdBWFC3N_eSmVcrLhk3yu5nJ9x_wC0ec6kVQ
[3]: https://github.com/apache/iceberg/blob/ed73ec43dd25c9023069ea1d3381a6d9229be53a/open-api/rest-catalog-open-api.yaml#L61
Re: [DISCUSS] REST: OAuth2 Authentication Guide
Posted by Yufei Gu <fl...@gmail.com>.
Thanks Christian. Nice write-up! Authentication is essential to a
production env. It's great to document it well given a lot of people don't
necessarily have enough OAthen2 knowledge. Looking forward to the doc PRs
and other client side changes.
Yufei
On Wed, Sep 18, 2024 at 8:31 AM Dmitri Bourlatchkov
<dm...@dremio.com.invalid> wrote:
> Hi Christian,
>
> Very nice proposal. Thanks for putting it together! I added some comments
> to the doc.
>
> I think it is related to PR #10753 [4], which proposes some foundational
> refactoring to the java REST client to enable further enhancements in
> OAuth2 flows.
>
> Cheers,
> Dmitri.
>
> [4] https://github.com/apache/iceberg/pull/10753
>
> On Wed, Sep 18, 2024 at 4:12 AM Christian Thiel
> <ch...@hansetag.com.invalid> wrote:
>
>> Dear everyone,
>>
>>
>> the Iceberg REST specification allows for different ways of
>> Authentication, OAuth2 is one of them. Until recently the OAuth2 /token
>> endpoint was part of the REST-spec together with datatypes required for the
>> client-credential flow. Both have since been removed from the spec for
>> security reasons [2].
>>
>> Probably because it was a part of the spec before, clients today
>> typically only implement the client-credential flow. This stays behind
>> OAuth2 capabilities and is unsuitable for human users. Common IdPs do not
>> implement the client-credential flow for principals of human users for good
>> reasons.
>>
>>
>>
>> To mitigate this problem, we propose an extension of the Iceberg
>> documentation in 3 steps. This proposal is neither an extension of the
>> Iceberg REST Catalog specification nor OAuth2 itself. The Iceberg REST
>> specification already specifies OAuth2 Authentication [3], which includes
>> all the flows mentioned in the document of this proposal [1].
>>
>>
>>
>> My proposal to go forward is as follows:
>>
>> 1. Use this proposals Google Doc for alignment in the community: [1]
>>
>> https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
>> Discuss in a catalog sync in 1-2 weeks.
>> 2. Condense consensus found in Google Doc to .md and add it to docs
>> 3. Implement additional flows in the iceberg-(java, python, rust ..)
>> packages.
>> For Java there is already a PR that goes in this direction which
>> could use some more attention:
>> https://github.com/apache/iceberg/pull/10753
>> For other languages I am not aware of any initiatives.
>> 4. Encourage clients to allow configuration of new flows for users
>>
>> Any feedback welcome!
>>
>> Thanks
>> - Christian
>>
>> [1]:
>> https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
>> [2]:
>> https://docs.google.com/document/d/1Xi5MRk8WdBWFC3N_eSmVcrLhk3yu5nJ9x_wC0ec6kVQ
>>
>> [3]:
>> https://github.com/apache/iceberg/blob/ed73ec43dd25c9023069ea1d3381a6d9229be53a/open-api/rest-catalog-open-api.yaml#L61
>>
>>
>>
>
Re: [DISCUSS] REST: OAuth2 Authentication Guide
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
Hi Christian,
Very nice proposal. Thanks for putting it together! I added some comments
to the doc.
I think it is related to PR #10753 [4], which proposes some foundational
refactoring to the java REST client to enable further enhancements in
OAuth2 flows.
Cheers,
Dmitri.
[4] https://github.com/apache/iceberg/pull/10753
On Wed, Sep 18, 2024 at 4:12 AM Christian Thiel
<ch...@hansetag.com.invalid> wrote:
> Dear everyone,
>
>
> the Iceberg REST specification allows for different ways of
> Authentication, OAuth2 is one of them. Until recently the OAuth2 /token
> endpoint was part of the REST-spec together with datatypes required for the
> client-credential flow. Both have since been removed from the spec for
> security reasons [2].
>
> Probably because it was a part of the spec before, clients today typically
> only implement the client-credential flow. This stays behind OAuth2
> capabilities and is unsuitable for human users. Common IdPs do not
> implement the client-credential flow for principals of human users for good
> reasons.
>
>
>
> To mitigate this problem, we propose an extension of the Iceberg
> documentation in 3 steps. This proposal is neither an extension of the
> Iceberg REST Catalog specification nor OAuth2 itself. The Iceberg REST
> specification already specifies OAuth2 Authentication [3], which includes
> all the flows mentioned in the document of this proposal [1].
>
>
>
> My proposal to go forward is as follows:
>
> 1. Use this proposals Google Doc for alignment in the community: [1]
>
> https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
> Discuss in a catalog sync in 1-2 weeks.
> 2. Condense consensus found in Google Doc to .md and add it to docs
> 3. Implement additional flows in the iceberg-(java, python, rust ..)
> packages.
> For Java there is already a PR that goes in this direction which could
> use some more attention: https://github.com/apache/iceberg/pull/10753
> For other languages I am not aware of any initiatives.
> 4. Encourage clients to allow configuration of new flows for users
>
> Any feedback welcome!
>
> Thanks
> - Christian
>
> [1]:
> https://docs.google.com/document/d/1A6bJfSzkTzDWUIegdckSsoaeFxZl1Qn5htI1jzyBQss/edit?usp=sharing
> [2]:
> https://docs.google.com/document/d/1Xi5MRk8WdBWFC3N_eSmVcrLhk3yu5nJ9x_wC0ec6kVQ
>
> [3]:
> https://github.com/apache/iceberg/blob/ed73ec43dd25c9023069ea1d3381a6d9229be53a/open-api/rest-catalog-open-api.yaml#L61
>
>
>