You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "astronautas (via GitHub)" <gi...@apache.org> on 2026/02/23 11:53:32 UTC
[I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
astronautas opened a new issue, #15414:
URL: https://github.com/apache/iceberg/issues/15414
### Apache Iceberg version
1.10.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Caused by: java.lang.IllegalStateException: OAuth2Credentials instance does not support refreshing the access token. An instance with a new access token should be used, or a derived type that supports refreshing.
Using GCSAuthManager. We assumed it would support credential refresh :/
@talatuyarer
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] closed issue #15414: GCSAuthManager does not seem to support credentials refresh - jobs crash mid
URL: https://github.com/apache/iceberg/issues/15414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "astronautas (via GitHub)" <gi...@apache.org>.
astronautas commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-4108625836
> Is it because you are using the OAuth token here explicitly in the config? I was also noticing similar issue with Flink. Wondering if it can work if we create an OAuth app for this flink job, and provide the app credentials directly as `credentials : <client_id>:<secret>` https://iceberg.apache.org/docs/nightly/configuration/#oauth2-auth-properties
No, we don't use an oauth token directly in the config. We use vended credentials. Actually getting rid of vended credentials i.e. running without refresh works just fine (e.g. using a k8s serviceAccount and then default application credentials get picked-up).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-6227287224
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "astronautas (via GitHub)" <gi...@apache.org>.
astronautas commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-3947466431
```
GCSFileIO also does support refreshing vended credentials, but it must be explicitly configured to use the
OAuth2RefreshCredentialsHandler. This handler knows how to call back to the catalog to fetch new credentials.
```
How to force use OAuth2RefreshCredentialsHandler? We use Lakekeeper
@c-thiel cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-3946861849
@astronautas Could you give more information about your setup. Access token are [valid for 1 hour](https://docs.cloud.google.com/docs/authentication/token-types#at-lifetime). If your job runs for more than an hour, it would suggest that the refresh mechanism is working. I am happy to try understand more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "astronautas (via GitHub)" <gi...@apache.org>.
astronautas commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-3946936067
> [@astronautas](https://github.com/astronautas) Could you give more information about your setup. Access token are [valid for 1 hour](https://docs.cloud.google.com/docs/authentication/token-types#at-lifetime). If your job runs for more than an hour, it would suggest that the refresh mechanism is working. I am happy to try understand more.
Indeed, our job runs for > 1h. We use Spark + Iceberg, and Lakekeeper, with GCSFileIO. The exception clearly states the refresh mechanism does not work, but I am not even sure where to look, which sort of dependency would be responsible for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-3947100239
@astronautas
GoogleAuthManager is used to authenticate the catalog session itself (e.g., via ADC or a specific service account). While it helps the catalog client authenticate with the REST Catalog, it does not automatically provide refresh capability for the data storage tokens (GCS tokens) vended by the catalog.
GCSFileIO also does support refreshing vended credentials, but it must be explicitly configured to use the
OAuth2RefreshCredentialsHandler. This handler knows how to call back to the catalog to fetch new credentials.
The catalog return something like below on LoadTableResponse. Which Catalog are you using ? BigLake ?
```json
{
"metadata": {
…
},
"metadata-location": "gs://<BUCKET_NAME>/test/test_tbl/metadata/010001-69708c21-0000-2f5a-a177-883d24f815b8.metadata.json",
"storage-credentials": [
{
"config": {
"expiration-time": "1771168951297",
"gcs.oauth2.refresh-credentials-enabled": "true",
"gcs.oauth2.refresh-credentials-endpoint": "v1/projects/<project-id>/catalogs/<Catalog-name>/namespaces/test/tables/test_tbl/credentials",
"gcs.oauth2.token": "<GCS OUTH2 TOKEN>",
"gcs.oauth2.token-expires-at": "1771168951297"
},
"prefix": "gs://<BUCKET_NAME>/test/test_tbl/"
}
]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "astronautas (via GitHub)" <gi...@apache.org>.
astronautas commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-3946959548
One important remark - the catalog in our case returns scoped credentials. Could it be that somehow the client `OAuth2Credentials instance` does not know how to refresh? We tried both Iceberg's GoogleAuthManager itself and without it, seems to break at `com.google.auth.oauth2.OAuth2Credentials`:
```
Caused by: java.lang.IllegalStateException: OAuth2Credentials instance does not support refreshing the access token. An instance with a new access token should be used, or a derived type that supports refreshing.
at com.google.auth.oauth2.OAuth2Credentials.refreshAccessToken(OAuth2Credentials.java:367)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:270)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:267)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at com.google.auth.oauth2.OAuth2Credentials$RefreshTask.run(OAuth2Credentials.java:644)
at org.apache.iceberg.gcp.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:591)
at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:233)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:183)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:98)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:199)
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
at com.google.cloud.storage.spi.v1.HttpStorageRpc$InvocationIdInitializer.initialize(HttpStorageRpc.java:187)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:91)
at com.google.api.client.http.HttpRequestFactory.buildPostRequest(HttpRequestFactory.java:128)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.open(HttpStorageRpc.java:1145)
at com.google.cloud.storage.ResumableMedia.lambda$startUploadForBlobInfo$0(ResumableMedia.java:38)
at com.google.cloud.storage.Retrying$DefaultRetrier.run(Retrying.java:165)
... 37 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-4277099064
Could you share an example of your Spark config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-6551074768
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "bharos (via GitHub)" <gi...@apache.org>.
bharos commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-4093624378
Is it because you are using the OAuth token here explicitly in the config?
I was also noticing similar issue with Flink.
Wondering if it can work if we create an OAuth app for this flink job, and provide the app credentials directly as
`credentials : <client_id>:<secret>`
[https://iceberg.apache.org/docs/nightly/configuration/#oauth2-auth-properties](https://iceberg.apache.org/docs/nightly/configuration/#oauth2-auth-properties)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] GCSAuthManager does not seem to support credentials refresh - jobs crash mid [iceberg]
Posted by "astronautas (via GitHub)" <gi...@apache.org>.
astronautas commented on issue #15414:
URL: https://github.com/apache/iceberg/issues/15414#issuecomment-3947483521
Could it be [this](https://github.com/apache/iceberg/issues/14634#issuecomment-3556342891)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org