You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Alexandre Dutra <ad...@apache.org> on 2026/04/24 13:28:27 UTC
[DISCUSS] REST spec change: passing arbitrary information to request signers
Hi all,
A while back, I submitted a proposal for the REST spec aimed at
formalizing a mechanism for circulating arbitrary properties from
catalog servers to request signer clients:
https://github.com/apache/iceberg/pull/15850
While the proposed modification is minimal, the review process has
stalled over whether to use "SHOULD" or "MUST" when defining the
property-passing requirements, which is why I am bringing this topic
to your attention.
In my view, the goal is to establish a clear contract where the REST
catalog server, the signer client, and the signer endpoint all commit
to these new rules. Consequently, I believe "MUST" is the more
suitable term.
Naturally, catalog servers following this new contract will need to
accommodate legacy clients. In my view, the specific handling of these
older clients is a matter of protocol misalignment mitigation
strategies and is not strictly relevant to the REST specification
itself. The goal of a protocol spec is to define what compliant
parties must do, not what non-compliant ones could do.
I would appreciate hearing your perspective on this.
Thanks,
Alex
Re: [DISCUSS] REST spec change: passing arbitrary information to request signers
Posted by Alexandre Dutra <ad...@apache.org>.
Hi Sung,
I appreciate your feedback! Starting with a simplified initial
structure sounds like a great plan.
Incorporating base-uri, endpoint-path, and properties is a solid
approach, especially since these are pre-existing concepts.
I'd also suggest including the headers field; although it's an
addition, I believe it is equally uncontroversial.
For the time being, we can exclude the more complex "security
primitives" (such as the signing token) until we have a more defined
refresh mechanism in place.
I went ahead and opened a PR for the above changes:
https://github.com/apache/iceberg/pull/16822
Thanks,
Alex
On Wed, Jun 10, 2026 at 7:50 PM Sung Yun <su...@apache.org> wrote:
>
> Hi Alex,
>
> Thanks for putting this together, and appreciate you summarizing the catalog sync outcomes so clearly.
>
> I really like the idea of having a struct with well-known properties. It’s helpful for broadly understood abstractions in the Spec and aligns nicely with the precedent set by StorageCredentials.
>
> Since defining a protocol to forward properties has been an ongoing discussion for a while [1], I wonder if it might be easier for us to break this work down into a few incremental steps.
>
> Given that we already have two of these that are required properties accepted in the IRC, what do you think about starting with a minimal initial structure containing just base-uri, endpoint-path, and properties? Since the other fields are optional, we could easily enhance the spec and add them later as the use cases and design are discussed.
>
> Sung
>
> [1] https://lists.apache.org/thread/gz5nm2xzlhzbc2y3sfossgflnkbm6vq5
>
> On 2026/05/27 13:30:30 Alexandre Dutra wrote:
> > Hi all,
> >
> > Gentle reminder: please chime in on the proposed spec changes.
> >
> > Thanks,
> > Alex
> >
> > On Wed, May 20, 2026 at 11:33 PM Alexandre Dutra <ad...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > Thanks for the good discussions today on this topic during the catalog
> > > sync meeting.
> > >
> > > Let me summarize the outcomes (please chime in if I didn't get this right):
> > >
> > > 1. The language should prefer SHOULD over MUST for the new features.
> > >
> > > 2. We don't want to keep using untyped property bags to implement this
> > > feature; instead, we need a proper structure under LoadTableResult to
> > > group all the settings to be communicated to the signer client, with
> > > clear usage semantics.
> > >
> > > 3. We want to distinguish security-related primitives from other data,
> > > so that such primitives could be properly refreshed since they are
> > > necessarily time-bounded.
> > >
> > > 4. We want any security primitive to be passed in a request header
> > > back to the signing endpoint. Headers are generally preferred over
> > > request body fields for sensitive information like this.
> > >
> > > 5. Deprecate the existing signer.uri, signer.endpoint table config
> > > properties, but these should still be honored for backwards
> > > compatibility.
> > >
> > > 6. The new structure should thus contain:
> > >
> > > 6.a. The signer URI (i.e. the server's address as seen by the client,
> > > taking into account X-Forward-* headers). This has precedence over the
> > > (now deprecated) signer.uri table property.
> > >
> > > 6.b. The signer endpoint (relative path to signer URI). This has
> > > precedence over the (now deprecated) signer.endpoint table property.
> > >
> > > 6.c. An optional "signing token", which should be sent in a
> > > well-known, predefined, Iceberg-specific header, e.g.
> > > "Iceberg-Remote-Sign-Authenticate" (tentative name). Ideally, such
> > > tokens should be refreshable.
> > >
> > > 6.d. [Optional] Other arbitrary static headers the signer should
> > > include in every request, and that never change.
> > >
> > > 6.e. [Optional] Other request properties the signer should include as
> > > is in every request, in the `properties` field of the request body;
> > > they never change.
> > >
> > > If the above requirements are correct, I have devised the following
> > > tentative OpenAPI schema:
> > >
> > > RemoteSigningConfig:
> > > description: |
> > > Configuration for the remote signer client.
> > > When present, clients MUST use this structure instead of the
> > > deprecated `signer.uri` and
> > > `signer.endpoint` properties in the `config` map.
> > > type: object
> > > properties:
> > > base-uri:
> > > type: string
> > > description: >
> > > The base URI of the signing service as perceived by the
> > > client, incorporating X-Forwarded-*
> > > headers set by proxies. When present, takes precedence over
> > > the deprecated `signer.uri`
> > > config property.
> > > endpoint-path:
> > > type: string
> > > description: >
> > > Relative path to be resolved against `base-uri` to form the
> > > full signing endpoint URI.
> > > When present, overrides the deprecated `signer.endpoint`
> > > config property.
> > > remote-signing-token:
> > > type: string
> > > description: >
> > > An optional, time-bounded remote signing token that signer
> > > clients SHOULD send as the
> > > `Iceberg-Remote-Sign-Authenticate` header in all requests to
> > > the signing endpoint.
> > > When absent, the server verifies the signer client's
> > > privileges on every call to the
> > > signing endpoint instead of relying on a pre-issued token.
> > > If the signing endpoint returns an
> > > `Iceberg-Remote-Sign-Authenticate` response header,
> > > the client MUST replace its current remote signing token with
> > > the new value.
> > > headers:
> > > allOf:
> > > - $ref: '#/components/schemas/MultiValuedMap'
> > > description: >
> > > Static headers the signer client SHOULD include unchanged in
> > > every request to the signing
> > > endpoint. These are non-security headers and do not change
> > > between requests.
> > > This field MUST NOT contain the
> > > `Iceberg-Remote-Sign-Authenticate` header name, which is
> > > reserved for the `token` field.
> > > properties:
> > > type: object
> > > additionalProperties:
> > > type: string
> > > description: >
> > > Static key-value pairs the signer client SHOULD pass through
> > > unchanged in the `properties`
> > > field of every `RemoteSignRequest` sent to the signing endpoint.
> > >
> > > Does the above suggestion align with what everybody had in mind? I
> > > used SHOULD as much as I could, even if I still think it feels
> > > awkward.
> > >
> > > Thanks,
> > > Alex
> > >
> > > On Fri, Apr 24, 2026 at 3:28 PM Alexandre Dutra <ad...@apache.org> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > A while back, I submitted a proposal for the REST spec aimed at
> > > > formalizing a mechanism for circulating arbitrary properties from
> > > > catalog servers to request signer clients:
> > > >
> > > > https://github.com/apache/iceberg/pull/15850
> > > >
> > > > While the proposed modification is minimal, the review process has
> > > > stalled over whether to use "SHOULD" or "MUST" when defining the
> > > > property-passing requirements, which is why I am bringing this topic
> > > > to your attention.
> > > >
> > > > In my view, the goal is to establish a clear contract where the REST
> > > > catalog server, the signer client, and the signer endpoint all commit
> > > > to these new rules. Consequently, I believe "MUST" is the more
> > > > suitable term.
> > > >
> > > > Naturally, catalog servers following this new contract will need to
> > > > accommodate legacy clients. In my view, the specific handling of these
> > > > older clients is a matter of protocol misalignment mitigation
> > > > strategies and is not strictly relevant to the REST specification
> > > > itself. The goal of a protocol spec is to define what compliant
> > > > parties must do, not what non-compliant ones could do.
> > > >
> > > > I would appreciate hearing your perspective on this.
> > > >
> > > > Thanks,
> > > > Alex
> >
Re: [DISCUSS] REST spec change: passing arbitrary information to request signers
Posted by Sung Yun <su...@apache.org>.
Hi Alex,
Thanks for putting this together, and appreciate you summarizing the catalog sync outcomes so clearly.
I really like the idea of having a struct with well-known properties. It’s helpful for broadly understood abstractions in the Spec and aligns nicely with the precedent set by StorageCredentials.
Since defining a protocol to forward properties has been an ongoing discussion for a while [1], I wonder if it might be easier for us to break this work down into a few incremental steps.
Given that we already have two of these that are required properties accepted in the IRC, what do you think about starting with a minimal initial structure containing just base-uri, endpoint-path, and properties? Since the other fields are optional, we could easily enhance the spec and add them later as the use cases and design are discussed.
Sung
[1] https://lists.apache.org/thread/gz5nm2xzlhzbc2y3sfossgflnkbm6vq5
On 2026/05/27 13:30:30 Alexandre Dutra wrote:
> Hi all,
>
> Gentle reminder: please chime in on the proposed spec changes.
>
> Thanks,
> Alex
>
> On Wed, May 20, 2026 at 11:33 PM Alexandre Dutra <ad...@apache.org> wrote:
> >
> > Hi all,
> >
> > Thanks for the good discussions today on this topic during the catalog
> > sync meeting.
> >
> > Let me summarize the outcomes (please chime in if I didn't get this right):
> >
> > 1. The language should prefer SHOULD over MUST for the new features.
> >
> > 2. We don't want to keep using untyped property bags to implement this
> > feature; instead, we need a proper structure under LoadTableResult to
> > group all the settings to be communicated to the signer client, with
> > clear usage semantics.
> >
> > 3. We want to distinguish security-related primitives from other data,
> > so that such primitives could be properly refreshed since they are
> > necessarily time-bounded.
> >
> > 4. We want any security primitive to be passed in a request header
> > back to the signing endpoint. Headers are generally preferred over
> > request body fields for sensitive information like this.
> >
> > 5. Deprecate the existing signer.uri, signer.endpoint table config
> > properties, but these should still be honored for backwards
> > compatibility.
> >
> > 6. The new structure should thus contain:
> >
> > 6.a. The signer URI (i.e. the server's address as seen by the client,
> > taking into account X-Forward-* headers). This has precedence over the
> > (now deprecated) signer.uri table property.
> >
> > 6.b. The signer endpoint (relative path to signer URI). This has
> > precedence over the (now deprecated) signer.endpoint table property.
> >
> > 6.c. An optional "signing token", which should be sent in a
> > well-known, predefined, Iceberg-specific header, e.g.
> > "Iceberg-Remote-Sign-Authenticate" (tentative name). Ideally, such
> > tokens should be refreshable.
> >
> > 6.d. [Optional] Other arbitrary static headers the signer should
> > include in every request, and that never change.
> >
> > 6.e. [Optional] Other request properties the signer should include as
> > is in every request, in the `properties` field of the request body;
> > they never change.
> >
> > If the above requirements are correct, I have devised the following
> > tentative OpenAPI schema:
> >
> > RemoteSigningConfig:
> > description: |
> > Configuration for the remote signer client.
> > When present, clients MUST use this structure instead of the
> > deprecated `signer.uri` and
> > `signer.endpoint` properties in the `config` map.
> > type: object
> > properties:
> > base-uri:
> > type: string
> > description: >
> > The base URI of the signing service as perceived by the
> > client, incorporating X-Forwarded-*
> > headers set by proxies. When present, takes precedence over
> > the deprecated `signer.uri`
> > config property.
> > endpoint-path:
> > type: string
> > description: >
> > Relative path to be resolved against `base-uri` to form the
> > full signing endpoint URI.
> > When present, overrides the deprecated `signer.endpoint`
> > config property.
> > remote-signing-token:
> > type: string
> > description: >
> > An optional, time-bounded remote signing token that signer
> > clients SHOULD send as the
> > `Iceberg-Remote-Sign-Authenticate` header in all requests to
> > the signing endpoint.
> > When absent, the server verifies the signer client's
> > privileges on every call to the
> > signing endpoint instead of relying on a pre-issued token.
> > If the signing endpoint returns an
> > `Iceberg-Remote-Sign-Authenticate` response header,
> > the client MUST replace its current remote signing token with
> > the new value.
> > headers:
> > allOf:
> > - $ref: '#/components/schemas/MultiValuedMap'
> > description: >
> > Static headers the signer client SHOULD include unchanged in
> > every request to the signing
> > endpoint. These are non-security headers and do not change
> > between requests.
> > This field MUST NOT contain the
> > `Iceberg-Remote-Sign-Authenticate` header name, which is
> > reserved for the `token` field.
> > properties:
> > type: object
> > additionalProperties:
> > type: string
> > description: >
> > Static key-value pairs the signer client SHOULD pass through
> > unchanged in the `properties`
> > field of every `RemoteSignRequest` sent to the signing endpoint.
> >
> > Does the above suggestion align with what everybody had in mind? I
> > used SHOULD as much as I could, even if I still think it feels
> > awkward.
> >
> > Thanks,
> > Alex
> >
> > On Fri, Apr 24, 2026 at 3:28 PM Alexandre Dutra <ad...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > A while back, I submitted a proposal for the REST spec aimed at
> > > formalizing a mechanism for circulating arbitrary properties from
> > > catalog servers to request signer clients:
> > >
> > > https://github.com/apache/iceberg/pull/15850
> > >
> > > While the proposed modification is minimal, the review process has
> > > stalled over whether to use "SHOULD" or "MUST" when defining the
> > > property-passing requirements, which is why I am bringing this topic
> > > to your attention.
> > >
> > > In my view, the goal is to establish a clear contract where the REST
> > > catalog server, the signer client, and the signer endpoint all commit
> > > to these new rules. Consequently, I believe "MUST" is the more
> > > suitable term.
> > >
> > > Naturally, catalog servers following this new contract will need to
> > > accommodate legacy clients. In my view, the specific handling of these
> > > older clients is a matter of protocol misalignment mitigation
> > > strategies and is not strictly relevant to the REST specification
> > > itself. The goal of a protocol spec is to define what compliant
> > > parties must do, not what non-compliant ones could do.
> > >
> > > I would appreciate hearing your perspective on this.
> > >
> > > Thanks,
> > > Alex
>
Re: [DISCUSS] REST spec change: passing arbitrary information to request signers
Posted by Alexandre Dutra <ad...@apache.org>.
Hi all,
Gentle reminder: please chime in on the proposed spec changes.
Thanks,
Alex
On Wed, May 20, 2026 at 11:33 PM Alexandre Dutra <ad...@apache.org> wrote:
>
> Hi all,
>
> Thanks for the good discussions today on this topic during the catalog
> sync meeting.
>
> Let me summarize the outcomes (please chime in if I didn't get this right):
>
> 1. The language should prefer SHOULD over MUST for the new features.
>
> 2. We don't want to keep using untyped property bags to implement this
> feature; instead, we need a proper structure under LoadTableResult to
> group all the settings to be communicated to the signer client, with
> clear usage semantics.
>
> 3. We want to distinguish security-related primitives from other data,
> so that such primitives could be properly refreshed since they are
> necessarily time-bounded.
>
> 4. We want any security primitive to be passed in a request header
> back to the signing endpoint. Headers are generally preferred over
> request body fields for sensitive information like this.
>
> 5. Deprecate the existing signer.uri, signer.endpoint table config
> properties, but these should still be honored for backwards
> compatibility.
>
> 6. The new structure should thus contain:
>
> 6.a. The signer URI (i.e. the server's address as seen by the client,
> taking into account X-Forward-* headers). This has precedence over the
> (now deprecated) signer.uri table property.
>
> 6.b. The signer endpoint (relative path to signer URI). This has
> precedence over the (now deprecated) signer.endpoint table property.
>
> 6.c. An optional "signing token", which should be sent in a
> well-known, predefined, Iceberg-specific header, e.g.
> "Iceberg-Remote-Sign-Authenticate" (tentative name). Ideally, such
> tokens should be refreshable.
>
> 6.d. [Optional] Other arbitrary static headers the signer should
> include in every request, and that never change.
>
> 6.e. [Optional] Other request properties the signer should include as
> is in every request, in the `properties` field of the request body;
> they never change.
>
> If the above requirements are correct, I have devised the following
> tentative OpenAPI schema:
>
> RemoteSigningConfig:
> description: |
> Configuration for the remote signer client.
> When present, clients MUST use this structure instead of the
> deprecated `signer.uri` and
> `signer.endpoint` properties in the `config` map.
> type: object
> properties:
> base-uri:
> type: string
> description: >
> The base URI of the signing service as perceived by the
> client, incorporating X-Forwarded-*
> headers set by proxies. When present, takes precedence over
> the deprecated `signer.uri`
> config property.
> endpoint-path:
> type: string
> description: >
> Relative path to be resolved against `base-uri` to form the
> full signing endpoint URI.
> When present, overrides the deprecated `signer.endpoint`
> config property.
> remote-signing-token:
> type: string
> description: >
> An optional, time-bounded remote signing token that signer
> clients SHOULD send as the
> `Iceberg-Remote-Sign-Authenticate` header in all requests to
> the signing endpoint.
> When absent, the server verifies the signer client's
> privileges on every call to the
> signing endpoint instead of relying on a pre-issued token.
> If the signing endpoint returns an
> `Iceberg-Remote-Sign-Authenticate` response header,
> the client MUST replace its current remote signing token with
> the new value.
> headers:
> allOf:
> - $ref: '#/components/schemas/MultiValuedMap'
> description: >
> Static headers the signer client SHOULD include unchanged in
> every request to the signing
> endpoint. These are non-security headers and do not change
> between requests.
> This field MUST NOT contain the
> `Iceberg-Remote-Sign-Authenticate` header name, which is
> reserved for the `token` field.
> properties:
> type: object
> additionalProperties:
> type: string
> description: >
> Static key-value pairs the signer client SHOULD pass through
> unchanged in the `properties`
> field of every `RemoteSignRequest` sent to the signing endpoint.
>
> Does the above suggestion align with what everybody had in mind? I
> used SHOULD as much as I could, even if I still think it feels
> awkward.
>
> Thanks,
> Alex
>
> On Fri, Apr 24, 2026 at 3:28 PM Alexandre Dutra <ad...@apache.org> wrote:
> >
> > Hi all,
> >
> > A while back, I submitted a proposal for the REST spec aimed at
> > formalizing a mechanism for circulating arbitrary properties from
> > catalog servers to request signer clients:
> >
> > https://github.com/apache/iceberg/pull/15850
> >
> > While the proposed modification is minimal, the review process has
> > stalled over whether to use "SHOULD" or "MUST" when defining the
> > property-passing requirements, which is why I am bringing this topic
> > to your attention.
> >
> > In my view, the goal is to establish a clear contract where the REST
> > catalog server, the signer client, and the signer endpoint all commit
> > to these new rules. Consequently, I believe "MUST" is the more
> > suitable term.
> >
> > Naturally, catalog servers following this new contract will need to
> > accommodate legacy clients. In my view, the specific handling of these
> > older clients is a matter of protocol misalignment mitigation
> > strategies and is not strictly relevant to the REST specification
> > itself. The goal of a protocol spec is to define what compliant
> > parties must do, not what non-compliant ones could do.
> >
> > I would appreciate hearing your perspective on this.
> >
> > Thanks,
> > Alex
Re: [DISCUSS] REST spec change: passing arbitrary information to request signers
Posted by Alexandre Dutra <ad...@apache.org>.
Hi all,
Thanks for the good discussions today on this topic during the catalog
sync meeting.
Let me summarize the outcomes (please chime in if I didn't get this right):
1. The language should prefer SHOULD over MUST for the new features.
2. We don't want to keep using untyped property bags to implement this
feature; instead, we need a proper structure under LoadTableResult to
group all the settings to be communicated to the signer client, with
clear usage semantics.
3. We want to distinguish security-related primitives from other data,
so that such primitives could be properly refreshed since they are
necessarily time-bounded.
4. We want any security primitive to be passed in a request header
back to the signing endpoint. Headers are generally preferred over
request body fields for sensitive information like this.
5. Deprecate the existing signer.uri, signer.endpoint table config
properties, but these should still be honored for backwards
compatibility.
6. The new structure should thus contain:
6.a. The signer URI (i.e. the server's address as seen by the client,
taking into account X-Forward-* headers). This has precedence over the
(now deprecated) signer.uri table property.
6.b. The signer endpoint (relative path to signer URI). This has
precedence over the (now deprecated) signer.endpoint table property.
6.c. An optional "signing token", which should be sent in a
well-known, predefined, Iceberg-specific header, e.g.
"Iceberg-Remote-Sign-Authenticate" (tentative name). Ideally, such
tokens should be refreshable.
6.d. [Optional] Other arbitrary static headers the signer should
include in every request, and that never change.
6.e. [Optional] Other request properties the signer should include as
is in every request, in the `properties` field of the request body;
they never change.
If the above requirements are correct, I have devised the following
tentative OpenAPI schema:
RemoteSigningConfig:
description: |
Configuration for the remote signer client.
When present, clients MUST use this structure instead of the
deprecated `signer.uri` and
`signer.endpoint` properties in the `config` map.
type: object
properties:
base-uri:
type: string
description: >
The base URI of the signing service as perceived by the
client, incorporating X-Forwarded-*
headers set by proxies. When present, takes precedence over
the deprecated `signer.uri`
config property.
endpoint-path:
type: string
description: >
Relative path to be resolved against `base-uri` to form the
full signing endpoint URI.
When present, overrides the deprecated `signer.endpoint`
config property.
remote-signing-token:
type: string
description: >
An optional, time-bounded remote signing token that signer
clients SHOULD send as the
`Iceberg-Remote-Sign-Authenticate` header in all requests to
the signing endpoint.
When absent, the server verifies the signer client's
privileges on every call to the
signing endpoint instead of relying on a pre-issued token.
If the signing endpoint returns an
`Iceberg-Remote-Sign-Authenticate` response header,
the client MUST replace its current remote signing token with
the new value.
headers:
allOf:
- $ref: '#/components/schemas/MultiValuedMap'
description: >
Static headers the signer client SHOULD include unchanged in
every request to the signing
endpoint. These are non-security headers and do not change
between requests.
This field MUST NOT contain the
`Iceberg-Remote-Sign-Authenticate` header name, which is
reserved for the `token` field.
properties:
type: object
additionalProperties:
type: string
description: >
Static key-value pairs the signer client SHOULD pass through
unchanged in the `properties`
field of every `RemoteSignRequest` sent to the signing endpoint.
Does the above suggestion align with what everybody had in mind? I
used SHOULD as much as I could, even if I still think it feels
awkward.
Thanks,
Alex
On Fri, Apr 24, 2026 at 3:28 PM Alexandre Dutra <ad...@apache.org> wrote:
>
> Hi all,
>
> A while back, I submitted a proposal for the REST spec aimed at
> formalizing a mechanism for circulating arbitrary properties from
> catalog servers to request signer clients:
>
> https://github.com/apache/iceberg/pull/15850
>
> While the proposed modification is minimal, the review process has
> stalled over whether to use "SHOULD" or "MUST" when defining the
> property-passing requirements, which is why I am bringing this topic
> to your attention.
>
> In my view, the goal is to establish a clear contract where the REST
> catalog server, the signer client, and the signer endpoint all commit
> to these new rules. Consequently, I believe "MUST" is the more
> suitable term.
>
> Naturally, catalog servers following this new contract will need to
> accommodate legacy clients. In my view, the specific handling of these
> older clients is a matter of protocol misalignment mitigation
> strategies and is not strictly relevant to the REST specification
> itself. The goal of a protocol spec is to define what compliant
> parties must do, not what non-compliant ones could do.
>
> I would appreciate hearing your perspective on this.
>
> Thanks,
> Alex