You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "nazq (via GitHub)" <gi...@apache.org> on 2025/11/06 17:08:48 UTC
[PR] feat(gcp): add support for external_account_authorized_user credentials [arrow-rs-object-store]
nazq opened a new pull request, #535:
URL: https://github.com/apache/arrow-rs-object-store/pull/535
## Summary
Adds support for `external_account_authorized_user` credentials used by Google Cloud Workforce Identity Federation. This credential type is created when users authenticate with `gcloud auth application-default login` using workforce identity pools with external identity providers.
## Problem
Previously, `object_store` only supported two Application Default Credential types:
- `service_account`
- `authorized_user`
Users with Workforce Identity Federation encountered deserialization errors when their ADC file contained `external_account_authorized_user` credentials, preventing them from using the library even though they had valid Google Cloud authentication.
This affected enterprise environments where organizations use external identity providers (like Azure AD, Okta, etc.) to authenticate users to Google Cloud resources.
## Solution
This PR extends credential support to include the `external_account_authorized_user` type:
1. **New credential struct** with OAuth2 fields (client_id, client_secret, refresh_token, token_url)
2. **TokenProvider implementation** using the STS (Security Token Service) OAuth token endpoint specified in the credential file
3. **Conversion to AuthorizedUserCredentials** for signing operations (same pattern as standard authorized_user)
4. **Builder integration** to handle the new credential type in selection logic
The implementation follows the same OAuth2 refresh flow as standard `authorized_user` credentials, but uses the custom `token_url` endpoint (typically `https://sts.googleapis.com/v1/oauthtoken`) specified in the credential file.
## Changes
**Files Modified:**
- `src/gcp/credential.rs`: Added ExternalAccountAuthorizedUserCredentials struct, enum variant, and TokenProvider implementation
- `src/gcp/builder.rs`: Updated credential selection logic to handle new type
- `src/gcp/mod.rs`: Added integration test
**Lines Changed:** +314 additions across 3 files
## Testing
### Unit Tests
- ✅ Deserialization with full JSON format
- ✅ Deserialization with minimal required fields
- ✅ Conversion to AuthorizedUserCredentials for signing
- ✅ Builder construction with sample credentials
### Integration Tests
- ✅ End-to-end test with put/get/delete operations (when TEST_INTEGRATION=1)
- ✅ Manual test with real ADC credentials (#[ignore])
**Test Results:**
```
test result: ok. 114 passed; 0 failed; 3 ignored
```
All existing tests continue to pass with no regressions.
## Credential Format Example
```json
{
"type": "external_account_authorized_user",
"audience": "//iam.googleapis.com/locations/global/workforcePools/pool/providers/provider",
"client_id": "xxxxx.apps.googleusercontent.com",
"client_secret": "secret",
"refresh_token": "token",
"token_url": "https://sts.googleapis.com/v1/oauthtoken",
"token_info_url": "https://sts.googleapis.com/v1/introspect",
"quota_project_id": "project-id"
}
```
## Impact
This change enables users in enterprise environments with Workforce Identity Federation to use `object_store` with their standard ADC configuration, matching the behavior of official Google Cloud client libraries (Python, Java, Go, etc.).
## References
- [Google Cloud Workforce Identity Federation](https://cloud.google.com/iam/docs/workforce-identity-federation)
- [Terraform Issue #34234](https://github.com/hashicorp/terraform/issues/34234) - Similar issue in Terraform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gcp): add support for external_account_authorized_user credentials [arrow-rs-object-store]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #535:
URL: https://github.com/apache/arrow-rs-object-store/pull/535#discussion_r2560052929
##########
src/gcp/mod.rs:
##########
@@ -450,4 +450,31 @@ mod test {
err
)
}
+
+ #[tokio::test]
+ async fn gcs_test_external_account_authorized_user_integration() {
+ maybe_skip_integration!();
+
+ // This test verifies that external_account_authorized_user credentials
Review Comment:
I don't understand how this test is verifying `external_account_authorized_user` -- it doesn't seem to configure such credentials 🤔
##########
src/gcp/builder.rs:
##########
@@ -746,4 +764,84 @@ mod tests {
panic!("{key} not propagated as ClientConfigKey");
}
}
+
+ #[test]
+ fn gcs_test_external_account_authorized_user_credentials() {
+ // Create an external_account_authorized_user credential file
+ // This format is used by workforce identity federation
+ let mut creds_file = NamedTempFile::new().unwrap();
+ creds_file
+ .write_all(
+ br#"{
+ "type": "external_account_authorized_user",
+ "audience": "//iam.googleapis.com/locations/global/workforcePools/test-pool/providers/test-provider",
+ "client_id": "test-client-id.apps.googleusercontent.com",
+ "client_secret": "test-client-secret",
+ "refresh_token": "test-refresh-token",
+ "token_url": "https://sts.googleapis.com/v1/oauthtoken",
+ "token_info_url": "https://sts.googleapis.com/v1/introspect",
+ "quota_project_id": "test-project"
+}"#,
+ )
+ .unwrap();
+
+ // Should successfully deserialize and create a builder
+ let result = GoogleCloudStorageBuilder::new()
+ .with_application_credentials(creds_file.path().to_str().unwrap())
+ .with_bucket_name("test-bucket")
+ .build();
+
+ // Build should succeed - the credentials are valid format
+ assert!(
+ result.is_ok(),
+ "Build should succeed with external_account_authorized_user credentials: {:?}",
+ result.err()
+ );
+ }
+
+ #[test]
+ #[ignore] // Only run manually when testing with real ADC
+ fn gcs_test_real_external_account_authorized_user_adc() {
Review Comment:
this test seems more like an example -- I wonder if it would be better as an example (or a no-run doc test)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gcp): add support for external_account_authorized_user credentials [arrow-rs-object-store]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #535:
URL: https://github.com/apache/arrow-rs-object-store/pull/535#issuecomment-3612074754
Marking as draft as I think this PR is no longer waiting on feedback and I am trying to make it easier to find PRs in need of review. Please mark it as ready for review when it is ready for another look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org