You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "SHIVANSH-ux-ys (via GitHub)" <gi...@apache.org> on 2026/09/01 09:43:04 UTC
[PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
SHIVANSH-ux-ys opened a new pull request, #17914:
URL: https://github.com/apache/iceberg/pull/17914
…(#17801)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#issuecomment-6810502019
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "adutra (via GitHub) via issues" <is...@iceberg.apache.org>.
adutra commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r4553182249
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -279,7 +281,7 @@ public SdkHttpFullRequest sign(
.build();
Key cacheKey = Key.from(remoteSigningRequest);
- SignedComponent cachedSignedComponent = SIGNED_COMPONENT_CACHE.getIfPresent(cacheKey);
+ SignedComponent cachedSignedComponent = signedComponentCache().getIfPresent(cacheKey);
Review Comment:
Not per sign request, but per `FileIO` instance. And we could have hundreds if not thousands of those in the same JVM.
A freshly-created Caffeine cache cost little memory, but if the cache is per `FileIO`, this means that 2 `FileIO`s belonging to the same principal cannot share the cache, which results in more calls to the sign endpoint than necessary.
Instead, I would keep the cache static, but would change the cache key.
But adding `token` or `client:secret` only to the cache key is not enough. Other auth managers may include other types of credentials, and since 1.12, catalog servers can also send credential-like data in the `remote-signing-config` object: these should be part of the cache key as well. As an extra protection, I think that the endpoint URI should also be part of the cache key.
In conclusion, the cache key should include the whole `properties()` map. Two clients with identical properties are indistinguishable to any signing server, and can safely keep sharing cache entries. The signing endpoint, the credentials and the remote signing configuration are all derived from them, and are thus included automatically.
Including the whole properties map doesn't hurt the cache efficiency either (or just marginally), because these properties are immutable for a given `FileIO` instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "singhpk234 (via GitHub) via issues" <is...@iceberg.apache.org>.
singhpk234 commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r4553812538
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -279,7 +281,7 @@ public SdkHttpFullRequest sign(
.build();
Key cacheKey = Key.from(remoteSigningRequest);
- SignedComponent cachedSignedComponent = SIGNED_COMPONENT_CACHE.getIfPresent(cacheKey);
+ SignedComponent cachedSignedComponent = signedComponentCache().getIfPresent(cacheKey);
Review Comment:
> Not per sign request, but per FileIO instance. And we could have hundreds if not thousands of those in the same JVM.
Ah i see its lazy annotated, thats fair
> Other auth managers may include other types of credentials, and since 1.12, catalog servers can also send credential-like data in the remote-signing-config
are we treating remote-signing-config a credential now ? my understanding is its opaque, and server ideally should revalisate that these belong to calling prinicipal / context before skipping the re-authZ ... what case you have in mind @adutra or you are just being cautious and want isolation to response as much as possible ?
+1 to the signing endpoint also part of the cache key ... i think we already do the same in AuthManager the same
https://github.com/apache/iceberg/blob/acea99d28a4be93a8c0dfc7bd8ece4a237324243/core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Manager.java#L200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "SHIVANSH-ux-ys (via GitHub)" <gi...@apache.org>.
SHIVANSH-ux-ys commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r3905946836
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -400,11 +400,19 @@ interface Key {
String uri();
- static Key from(RemoteSignRequest request) {
+ @Nullable
+ String credential();
+
+ @Nullable
+ String token();
+
+ static Key from(RemoteSignRequest request, S3V4RestSignerClient signerClient) {
return ImmutableKey.builder()
.method(request.method())
.region(request.region())
.uri(request.uri().toString())
+ .credential(signerClient.credential())
+ .token(signerClient.token().get())
Review Comment:
Thanks for reviewing @singhpk234!
Makes sense - updated Key.from to prioritize token if set and fallback to credential. Also added a unit test (testSignedComponentCacheKeyIsolation) in TestS3V4RestSignerClient.java to verify cache key isolation across different credentials.
Pushed the updates!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "ShivanshhhG (via GitHub)" <gi...@apache.org>.
ShivanshhhG commented on PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#issuecomment-5991160828
Hi @yangshangqing95,
Following up on your review feedback:
I have updated `doesNotReuseSignedComponentAcrossSignerClients` into a full end-to-end behavioral regression test:
1. Client 1 issues a sign request for an S3 resource; the mock signer returns a cacheable response (`Cache-Control: private`) with signature `AWS4-HMAC-SHA256 sig-client-1`.
2. A repeated sign request through Client 1 hits its local cache without dispatching an additional remote sign request (`callCount == 1`).
3. Client 2 (representing a separate principal / client instance) issues an identical request (same HTTP method, AWS region, and URI). It verifies that Client 2 calls its own signer, does not reuse Client 1's cached signature, and receives `AWS4-HMAC-SHA256 sig-client-2` (`callCount == 2`).
4. Subsequent calls on Client 2 hit Client 2's cache.
5. In addition, `assertThat(client1.signedComponentCache()).isNotSameAs(client2.signedComponentCache())` verifies structural cache instance isolation.
All tests and spotless checks pass cleanly. Ready for another review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r3907069510
##########
aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3V4RestSignerClient.java:
##########
@@ -233,4 +237,42 @@ public static Stream<Arguments> legacySignerProperties() {
"https://legacy-signer.com",
"https://legacy-signer.com/v1/legacy/sign"));
}
+
+ @Test
+ void testSignedComponentCacheKeyIsolation() throws Exception {
Review Comment:
Nit: Iceberg's current convention for newly added tests avoids the `test` prefix, so this could be `signedComponentCacheKeyIsolation()` or, if converted to a behavior level regression test, something like `doesNotReuseSignedComponentAcrossSignerClients()`.
##########
aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3V4RestSignerClient.java:
##########
@@ -233,4 +237,42 @@ public static Stream<Arguments> legacySignerProperties() {
"https://legacy-signer.com",
"https://legacy-signer.com/v1/legacy/sign"));
}
+
+ @Test
+ void testSignedComponentCacheKeyIsolation() throws Exception {
+ Map<String, String> properties1 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.CREDENTIAL,
+ "user1:secret1");
+ Map<String, String> properties2 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.CREDENTIAL,
+ "user2:secret2");
+
+ try (S3V4RestSignerClient client1 =
+ ImmutableS3V4RestSignerClient.builder().properties(properties1).build();
+ S3V4RestSignerClient client2 =
+ ImmutableS3V4RestSignerClient.builder().properties(properties2).build()) {
+ RemoteSignRequest request =
+ ImmutableRemoteSignRequest.builder()
+ .method("GET")
+ .region("us-east-1")
+ .uri(URI.create("https://bucket.s3.amazonaws.com/key"))
+ .provider(S3V4RestSignerClient.S3_PROVIDER)
+ .build();
+
+ S3V4RestSignerClient.Key key1 = S3V4RestSignerClient.Key.from(request, client1);
+ S3V4RestSignerClient.Key key2 = S3V4RestSignerClient.Key.from(request, client2);
+
+ assertThat(key1).isNotEqualTo(key2);
Review Comment:
Could we make this a regression test for the actual cache behavior rather than testing Key.equals directly?
For #17801, I'd expect the test to populate a cacheable signed response through signer/client A, then make the same method/region/URI request through signer/client B and verify that B still calls its signer and receives B's signed component. That directly protects the security invariant and doesn't couple the test to the current cache-key implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "ShivanshhhG (via GitHub)" <gi...@apache.org>.
ShivanshhhG commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r4246568118
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -400,11 +400,19 @@ interface Key {
String uri();
- static Key from(RemoteSignRequest request) {
+ @Nullable
+ String credential();
+
+ @Nullable
+ String token();
+
+ static Key from(RemoteSignRequest request, S3V4RestSignerClient signerClient) {
return ImmutableKey.builder()
.method(request.method())
.region(request.region())
.uri(request.uri().toString())
+ .credential(signerClient.credential())
+ .token(signerClient.token().get())
Review Comment:
Thanks @yangshangqing95! That made total sense - I adopted your suggestion and switched `signedComponentCache` to be a per-instance `@Value.Lazy` cache on `S3V4RestSignerClient`.
That cleanly isolates the cache structurally per client instance without needing to store credentials/tokens in a static key. Also added an end-to-end regression test (`doesNotReuseSignedComponentAcrossSignerClients`) verifying that two separate client instances don't share cached signatures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "singhpk234 (via GitHub) via issues" <is...@iceberg.apache.org>.
singhpk234 commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r4545654548
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -279,7 +281,7 @@ public SdkHttpFullRequest sign(
.build();
Key cacheKey = Key.from(remoteSigningRequest);
- SignedComponent cachedSignedComponent = SIGNED_COMPONENT_CACHE.getIfPresent(cacheKey);
+ SignedComponent cachedSignedComponent = signedComponentCache().getIfPresent(cacheKey);
Review Comment:
wouldn't this create a cache obj per sign request?
i think what we need to is just make the `token` which would be used as bearer or `client:secret` as part of the cache key from the property
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "ShivanshhhG (via GitHub) via issues" <is...@iceberg.apache.org>.
ShivanshhhG commented on PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#issuecomment-6888338882
Thanks @adutra and @singhpk234! I have updated the PR following the agreed approach:
1. Retained the static, process-wide `SIGNED_COMPONENT_CACHE` across `FileIO` instances.
2. Included the full `properties()` map in the cache `Key`, ensuring that all credential/auth/remote-signing-config and endpoint variations naturally isolate cache entries, while clients sharing identical properties safely share cached signatures.
3. Restored the case-insensitive `Cache-Control` handling and added test coverage verifying both cross-principal isolation and cache sharing across distinct client instances with identical properties.
4. Verified that all unit tests and Spotless checks in `:iceberg-aws` pass cleanly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "SHIVANSH-ux-ys (via GitHub)" <gi...@apache.org>.
SHIVANSH-ux-ys commented on PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#issuecomment-5498649546
Thanks @yangshangqing95!
Refactored `signedComponentCache` to be a per-instance `@Value.Lazy` cache on `S3V4RestSignerClient` rather than a shared static cache. This structurally isolates signed components per client instance without storing credentials in the cache key.
Also updated the test name to `doesNotReuseSignedComponentAcrossSignerClients()` following Iceberg's naming convention and pushed the updates!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "singhpk234 (via GitHub) via issues" <is...@iceberg.apache.org>.
singhpk234 commented on PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#issuecomment-6812996574
cc @adutra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "adutra (via GitHub) via issues" <is...@iceberg.apache.org>.
adutra commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r4554009354
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -279,7 +281,7 @@ public SdkHttpFullRequest sign(
.build();
Key cacheKey = Key.from(remoteSigningRequest);
- SignedComponent cachedSignedComponent = SIGNED_COMPONENT_CACHE.getIfPresent(cacheKey);
+ SignedComponent cachedSignedComponent = signedComponentCache().getIfPresent(cacheKey);
Review Comment:
> are we treating remote-signing-config a credential now ?
Not "credential" necessarily, but at least like something that must be part of the cache key. We don't known what it contains and whether a specific server sends information there that is relevant for the signing operation; mixing cache entries that contain different remote-signing-configs is therefore risky.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r3907034694
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -400,11 +400,19 @@ interface Key {
String uri();
- static Key from(RemoteSignRequest request) {
+ @Nullable
+ String credential();
+
+ @Nullable
+ String token();
+
+ static Key from(RemoteSignRequest request, S3V4RestSignerClient signerClient) {
return ImmutableKey.builder()
.method(request.method())
.region(request.region())
.uri(request.uri().toString())
+ .credential(signerClient.credential())
+ .token(signerClient.token().get())
Review Comment:
I agree with using either the token or the credential rather than both. My remaining concern is whether either value alone is sufficient to define the principal boundary for a process-wide cache.
The auth session is derived from more than these two values (AuthManager, OAuth server URI, scope/resource/audience, and potentially non-OAuth auth types). For example, two signer clients can use the same credential against different OAuth servers/resources, while non-OAuth principals may have both token and credential unset. Those clients would still produce the same cache key here.
Would it be simpler and safer to scope the signed-component cache to the `S3V4RestSignerClient` instance instead? That makes the isolation structural rather than trying to encode every authentication dimension into Key, and it also avoids retaining bearer tokens/client secrets in a process-wide cache key.
FWIW, #16524 has already been updated to make this cache per-instance, so it may be worth coordinating the two fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#issuecomment-5963600670
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r3904897121
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -400,11 +400,19 @@ interface Key {
String uri();
- static Key from(RemoteSignRequest request) {
+ @Nullable
+ String credential();
+
+ @Nullable
+ String token();
+
+ static Key from(RemoteSignRequest request, S3V4RestSignerClient signerClient) {
return ImmutableKey.builder()
.method(request.method())
.region(request.region())
.uri(request.uri().toString())
+ .credential(signerClient.credential())
+ .token(signerClient.token().get())
Review Comment:
it should ideally either credential or token right not both because of case when credentials is exchanged for token ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Isolate S3V4RestSignerClient signed request cache per principal … [iceberg]
Posted by "adutra (via GitHub) via issues" <is...@iceberg.apache.org>.
adutra commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r4584280181
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -380,11 +380,14 @@ interface Key {
String uri();
- static Key from(RemoteSignRequest request) {
+ Map<String, String> properties();
+
+ static Key from(RemoteSignRequest request, Map<String, String> properties) {
return ImmutableKey.builder()
.method(request.method())
.region(request.region())
.uri(request.uri().toString())
+ .properties(properties)
Review Comment:
I would rather distinguish the request properties from the client properties. Someone could in theory override `requestPropertiesSupplier`, even if that's unlikely.
```suggestion
Map<String, String> requestProperties();
Map<String, String> clientProperties();
static Key from(RemoteSignRequest request, Map<String, String> clientProperties) {
return ImmutableKey.builder()
.method(request.method())
.region(request.region())
.uri(request.uri().toString())
.requestProperties(request.properties())
.clientProperties(clientProperties)
```
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
Review Comment:
I would add a javadoc here to explain the cache key logic, which imho is not easy to grasp:
```suggestion
/**
* The key of the signed component cache. Besides the request itself, it includes the properties
* of the signer client, since they determine how the request is signed: signing endpoint,
* authentication, and remote signing configuration. The request properties are included as well,
* since they do not necessarily derive from the client properties when {@link
* S3V4RestSignerClient#requestPropertiesSupplier()} is overridden.
*/
@Value.Immutable
```
##########
aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3V4RestSignerClient.java:
##########
@@ -262,4 +265,105 @@ public static Stream<Arguments> validOAuth2Properties() {
"custom",
"token"));
}
+
+ @Test
+ void doesNotReuseSignedComponentAcrossSignerClients() throws Exception {
+ Map<String, String> properties1 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token1");
+ Map<String, String> properties2 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token2");
+
+ URI requestUri = URI.create("https://bucket.s3.us-west-2.amazonaws.com/object-key");
+ SdkHttpFullRequest request =
+ SdkHttpFullRequest.builder()
+ .method(SdkHttpMethod.GET)
+ .uri(requestUri)
+ .protocol("https")
+ .host("bucket.s3.us-west-2.amazonaws.com")
+ .encodedPath("/object-key")
+ .build();
+
+ ExecutionAttributes executionAttributes = new ExecutionAttributes();
+ executionAttributes.putAttribute(AwsSignerExecutionAttribute.SERVICE_SIGNING_NAME, "s3");
Review Comment:
You may want to annotate this test with `@SuppressWarnings("deprecation")` to suppress the warnings in lines 299-302.
##########
aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3V4RestSignerClient.java:
##########
@@ -262,4 +265,105 @@ public static Stream<Arguments> validOAuth2Properties() {
"custom",
"token"));
}
+
+ @Test
+ void doesNotReuseSignedComponentAcrossSignerClients() throws Exception {
+ Map<String, String> properties1 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token1");
+ Map<String, String> properties2 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token2");
+
+ URI requestUri = URI.create("https://bucket.s3.us-west-2.amazonaws.com/object-key");
+ SdkHttpFullRequest request =
+ SdkHttpFullRequest.builder()
+ .method(SdkHttpMethod.GET)
+ .uri(requestUri)
+ .protocol("https")
+ .host("bucket.s3.us-west-2.amazonaws.com")
+ .encodedPath("/object-key")
+ .build();
+
+ ExecutionAttributes executionAttributes = new ExecutionAttributes();
+ executionAttributes.putAttribute(AwsSignerExecutionAttribute.SERVICE_SIGNING_NAME, "s3");
+ executionAttributes.putAttribute(AwsSignerExecutionAttribute.SIGNING_REGION, Region.US_WEST_2);
+ executionAttributes.putAttribute(
+ AwsSignerExecutionAttribute.AWS_CREDENTIALS,
+ AnonymousCredentialsProvider.create().resolveCredentials());
+
+ AtomicInteger callCount = new AtomicInteger();
+ when(S3V4RestSignerClient.httpClient.post(
+ Mockito.eq("https://signer.com/v1/sign"),
+ Mockito.any(RemoteSignRequest.class),
+ Mockito.eq(RemoteSignResponse.class),
+ Mockito.anyMap(),
+ Mockito.any(),
+ Mockito.any()))
+ .thenAnswer(
+ invocation -> {
+ int count = callCount.incrementAndGet();
+ Consumer<Map<String, String>> headersConsumer = invocation.getArgument(5);
+ if (headersConsumer != null) {
+ headersConsumer.accept(Map.of("Cache-Control", "private"));
+ }
+ return ImmutableRemoteSignResponse.builder()
+ .uri(requestUri)
+ .headers(Map.of("Authorization", List.of("AWS4-HMAC-SHA256 sig-client-" + count)))
+ .build();
+ });
+
+ try (S3V4RestSignerClient client1 =
+ ImmutableS3V4RestSignerClient.builder().properties(properties1).build();
+ S3V4RestSignerClient client2 =
+ ImmutableS3V4RestSignerClient.builder().properties(properties2).build()) {
+
+ // Client 1 makes request: cache miss, calls signer, response is cached
+ SdkHttpFullRequest signed1 = client1.sign(request, executionAttributes);
+ assertThat(signed1.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-1");
+ assertThat(callCount.get()).isEqualTo(1);
+
+ // Client 1 repeated request: cache hit, does not call signer again
+ SdkHttpFullRequest signed1Repeat = client1.sign(request, executionAttributes);
+ assertThat(signed1Repeat.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-1");
+ assertThat(callCount.get()).isEqualTo(1);
+
+ // Client 2 makes the same request: must NOT reuse Client 1's cached signature.
+ // It must call signer and receive Client 2's own signature.
+ SdkHttpFullRequest signed2 = client2.sign(request, executionAttributes);
+ assertThat(signed2.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-2");
+ assertThat(callCount.get()).isEqualTo(2);
+
+ // Client 2 repeated request: cache hit on client 2's cache
+ SdkHttpFullRequest signed2Repeat = client2.sign(request, executionAttributes);
+ assertThat(signed2Repeat.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-2");
+ assertThat(callCount.get()).isEqualTo(2);
+
+ // Verify that another client instance with identical properties shares the cache (e.g.
+ // multiple FileIOs for the same principal)
+ try (S3V4RestSignerClient client3 =
+ ImmutableS3V4RestSignerClient.builder().properties(properties1).build()) {
+ SdkHttpFullRequest signed3 = client3.sign(request, executionAttributes);
+ assertThat(signed3.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-1");
+ assertThat(callCount.get())
+ .as("Client 3 shares identical properties with Client 1 and must hit the cache")
+ .isEqualTo(2);
+ }
+ }
+ }
Review Comment:
I would add two more tests to test that the endpoint and `remote-signing-config` are also part of the cache key:
```java
@Test
void signedComponentCacheKeyIncludesRemoteSigningConfig() throws Exception {
URI uri = URI.create("https://bucket.s3.us-west-2.amazonaws.com/" + UUID.randomUUID());
AtomicInteger signRequests = mockSigningServer(uri);
SdkHttpFullRequest request =
SdkHttpFullRequest.builder().uri(uri).method(SdkHttpMethod.GET).build();
try (S3V4RestSignerClient client1 = signerClient("https://signer.com", "value-1");
S3V4RestSignerClient client2 = signerClient("https://signer.com", "value-2")) {
client1.sign(request, executionAttributes());
client2.sign(request, executionAttributes());
}
assertThat(signRequests.get())
.as("requests with different remote signing configs should be signed separately")
.isEqualTo(2);
}
@Test
void signedComponentCacheKeyIncludesSigningEndpoint() throws Exception {
URI uri = URI.create("https://bucket.s3.us-west-2.amazonaws.com/" + UUID.randomUUID());
AtomicInteger signRequests = mockSigningServer(uri);
SdkHttpFullRequest request =
SdkHttpFullRequest.builder().uri(uri).method(SdkHttpMethod.GET).build();
try (S3V4RestSignerClient client1 = signerClient("https://signer-1.com", "value");
S3V4RestSignerClient client2 = signerClient("https://signer-2.com", "value")) {
client1.sign(request, executionAttributes());
client2.sign(request, executionAttributes());
}
assertThat(signRequests.get())
.as("requests to different signing endpoints should be signed separately")
.isEqualTo(2);
}
private static AtomicInteger mockSigningServer(URI uri) {
AtomicInteger signRequests = new AtomicInteger();
when(S3V4RestSignerClient.httpClient.post(
Mockito.anyString(),
Mockito.any(),
Mockito.eq(RemoteSignResponse.class),
Mockito.anyMap(),
Mockito.any(),
Mockito.any()))
.thenAnswer(
invocation -> {
signRequests.incrementAndGet();
Consumer<Map<String, String>> responseHeaders = invocation.getArgument(5);
responseHeaders.accept(Map.of("Cache-Control", "private"));
return ImmutableRemoteSignResponse.builder()
.uri(uri)
.headers(Map.of("Authorization", List.of("AWS4-HMAC-SHA256 Credential=key")))
.build();
});
return signRequests;
}
@SuppressWarnings("deprecation")
private static ExecutionAttributes executionAttributes() {
return ExecutionAttributes.builder()
.put(
AwsSignerExecutionAttribute.AWS_CREDENTIALS,
AwsBasicCredentials.create("accessKeyId", "secretAccessKey"))
.put(AwsSignerExecutionAttribute.SIGNING_REGION, Region.US_WEST_2)
.put(AwsSignerExecutionAttribute.SERVICE_SIGNING_NAME, "s3")
.build();
}
private static S3V4RestSignerClient signerClient(String signerUri, String propertyValue) {
return ImmutableS3V4RestSignerClient.builder()
.properties(
Map.of(
CatalogProperties.URI,
signerUri,
RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
"v1/namespaces/ns1/tables/t1/sign",
RESTCatalogProperties.REMOTE_SIGNING_CONFIG,
RemoteSigningConfigParser.toJson(
ImmutableRemoteSigningConfig.builder()
.putProperties("property", propertyValue)
.build()),
OAuth2Properties.TOKEN,
"token"))
.build();
}
```
##########
aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3V4RestSignerClient.java:
##########
@@ -262,4 +265,105 @@ public static Stream<Arguments> validOAuth2Properties() {
"custom",
"token"));
}
+
+ @Test
+ void doesNotReuseSignedComponentAcrossSignerClients() throws Exception {
+ Map<String, String> properties1 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token1");
+ Map<String, String> properties2 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token2");
+
+ URI requestUri = URI.create("https://bucket.s3.us-west-2.amazonaws.com/object-key");
+ SdkHttpFullRequest request =
+ SdkHttpFullRequest.builder()
+ .method(SdkHttpMethod.GET)
+ .uri(requestUri)
+ .protocol("https")
+ .host("bucket.s3.us-west-2.amazonaws.com")
+ .encodedPath("/object-key")
+ .build();
Review Comment:
Let's use a random object key for increased cross-test safety:
```suggestion
String objectKey = UUID.randomUUID().toString();
URI requestUri = URI.create("https://bucket.s3.us-west-2.amazonaws.com/" + objectKey);
SdkHttpFullRequest request =
SdkHttpFullRequest.builder()
.method(SdkHttpMethod.GET)
.uri(requestUri)
.protocol("https")
.host("bucket.s3.us-west-2.amazonaws.com")
.encodedPath("/" + objectKey)
.build();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org