You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "RobinLin666 (via GitHub)" <gi...@apache.org> on 2024/09/11 13:28:14 UTC
[PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
RobinLin666 opened a new pull request, #6382:
URL: https://github.com/apache/arrow-rs/pull/6382
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
In Azure Fabric, we use token service to get user access token, for supporting long reading and writing operation and auto refresh access token, we implement this.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
This pull request introduces significant enhancements to the Azure integration within the `object_store` module, including the addition of the `jsonwebtoken` dependency and the implementation of a new `FabricTokenOAuthProvider`. These changes aim to improve the authentication mechanism and add support for fabric token services.
### Dependency Updates:
* Added `jsonwebtoken` as an optional dependency to `Cargo.toml`. (`object_store/Cargo.toml` [object_store/Cargo.tomlR58-R65](diffhunk://#diff-3b95b70dfe3a83b11c60164ca74b3c8eb556a24cdba0a3d6dd9f71e612d3f50dR58-R65))
### Azure Builder Enhancements:
* Added new fields to `MicrosoftAzureBuilder` to support fabric token services, including `fabric_token_service_url`, `fabric_workload_host`, `fabric_session_token`, and `fabric_cluster_identifier`. (`object_store/src/azure/builder.rs` [object_store/src/azure/builder.rsR175-R182](diffhunk://#diff-8a210e33eba6ee7a3f8413616b7737e5f50658e481e1707024ab91b97e106363R175-R182))
* Updated `AzureConfigKey` to include new configuration keys related to fabric token services. (`object_store/src/azure/builder.rs` [object_store/src/azure/builder.rsR347-R374](diffhunk://#diff-8a210e33eba6ee7a3f8413616b7737e5f50658e481e1707024ab91b97e106363R347-R374))
* Modified `impl AsRef<str>` and `impl FromStr` for `AzureConfigKey` to handle new fabric token service keys. (`object_store/src/azure/builder.rs` [[1]](diffhunk://#diff-8a210e33eba6ee7a3f8413616b7737e5f50658e481e1707024ab91b97e106363R400-R403) [[2]](diffhunk://#diff-8a210e33eba6ee7a3f8413616b7737e5f50658e481e1707024ab91b97e106363R449-R456)
* Enhanced `MicrosoftAzureBuilder` to set and get the new fabric token service-related fields. (`object_store/src/azure/builder.rs` [[1]](diffhunk://#diff-8a210e33eba6ee7a3f8413616b7737e5f50658e481e1707024ab91b97e106363R576-R583) [[2]](diffhunk://#diff-8a210e33eba6ee7a3f8413616b7737e5f50658e481e1707024ab91b97e106363R620-R623)
* Added logic to `MicrosoftAzureBuilder` to create a `FabricTokenOAuthProvider` if fabric token service fields are provided. (`object_store/src/azure/builder.rs` [object_store/src/azure/builder.rsR919-R942](diffhunk://#diff-8a210e33eba6ee7a3f8413616b7737e5f50658e481e1707024ab91b97e106363R919-R942))
### Credential Enhancements:
* Introduced `FabricTokenOAuthProvider` to handle OAuth token challenges for fabric token services, including methods for token validation and fetching. (`object_store/src/azure/credential.rs` [object_store/src/azure/credential.rsR943-R1049](diffhunk://#diff-53498fbed9bb6cfcf50c4c9335b23cd5add9d77be3a150dca7235a05716718b4R943-R1049))
* Added new headers and constants related to fabric token services. (`object_store/src/azure/credential.rs` [object_store/src/azure/credential.rsR55-R63](diffhunk://#diff-53498fbed9bb6cfcf50c4c9335b23cd5add9d77be3a150dca7235a05716718b4R55-R63))
These changes collectively enhance the Azure integration by supporting more complex authentication mechanisms, particularly for environments utilizing fabric token services.
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382#issuecomment-2365132985
Thank you very much @tustvold @RobinLin666 and @roeap 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "RobinLin666 (via GitHub)" <gi...@apache.org>.
RobinLin666 commented on PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382#issuecomment-2360274919
Thanks all, please help to merge the PR if no question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382#issuecomment-2360840961
Let's wait a day or two before merging to see if @roeap has some time to review. This is getting very close.
Thanks for your patience @RobinLin666 and the help @tustvold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "RobinLin666 (via GitHub)" <gi...@apache.org>.
RobinLin666 commented on PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382#issuecomment-2364797934
Thanks @roeap , yes, I have validated in Fabric Notebook.
<img width="1312" alt="image" src="https://github.com/user-attachments/assets/176c6af4-f47e-41da-8c2e-b545463d7688">
(For testing, I printed something, and deleted it in the PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382#issuecomment-2345309966
https://jwt.io/introduction you should be able to simply split the string, base64 decode the middle chunk and parse the JSON
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382#discussion_r1769527513
##########
object_store/src/azure/builder.rs:
##########
@@ -336,6 +344,34 @@ pub enum AzureConfigKey {
/// - `disable_tagging`
DisableTagging,
+ /// Fabric token service url
Review Comment:
I double checked and this enum is marked `#[non_exhaustive]` and thus it is ok to add new variants without breaking the API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "RobinLin666 (via GitHub)" <gi...@apache.org>.
RobinLin666 commented on PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382#issuecomment-2345201073
> The JWT logic seems a little odd to me, are we just using it to decode the expiry? If so could we avoid the additional dependency?
Hi @tustvold Thank you for you review. Yes, because Token Service only returns a JWT token, so I need to decode the expiry. Any advice without dependency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382#issuecomment-2344095609
The JWT logic seems a little odd to me, are we just using it to decode the expiry? If so could we avoid the additional dependency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] object_score: Support Azure Fabric OAuth Provider [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6382:
URL: https://github.com/apache/arrow-rs/pull/6382#issuecomment-2359326341
I am depressed about the large review backlog in this crate. We are looking for more help from the community reviewing PRs -- see https://github.com/apache/arrow-rs/issues/6418 for more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org