You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "stevenzwu (via GitHub)" <gi...@apache.org> on 2026/06/02 05:01:12 UTC

Re: [PR] [SPEC] Add finer grained read restrictions as part of loadTable [iceberg]

stevenzwu commented on code in PR #13879:
URL: https://github.com/apache/iceberg/pull/13879#discussion_r3338744689


##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3480,6 +3480,341 @@ components:
           additionalProperties:
             type: string
 
+    ReadRestrictions:
+      type: object
+      description: >
+          Read restrictions for a table, including column projections and row filter expressions.
+
+          A reader MUST enforce the restrictions defined in this object when reading data
+          from the table. Read restrictions returned in a loadTable response apply to every

Review Comment:
   The earlier "trusted clients?" thread on this section converged on the model "older client = untrusted, admin manages trust" — but the spec text here still reads `A reader MUST enforce` without that qualification. Suggest folding the resolution into the prose, e.g.:
   
   > A reader that the catalog has authorized to enforce read restrictions MUST enforce them when reading data from the table. Catalogs SHOULD NOT include `ReadRestrictions` in responses to readers whose support has not been established. Mechanisms for establishing that trust (e.g., mTLS, OAuth on-behalf-of) are out of scope for this spec.
   
   Otherwise, a strictly-conforming reader has no signal that this is opt-in, and a strictly-conforming catalog has no signal that older readers should not receive this field.
   



##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3480,6 +3480,309 @@ components:
           additionalProperties:
             type: string
 
+    ReadRestrictions:
+      type: object
+      description: >
+          Read restrictions for a table, including column projections and row filter expressions.
+
+          A client MUST enforce the restrictions defined in this object when reading data
+          from the table.
+
+          These restrictions apply only to the authenticated principal, user, or account
+          associated with the request. They MUST NOT be interpreted as global policy and
+          MUST NOT be applied beyond the entity identified by the Authentication header
+          (or other applicable authentication mechanism).
+
+          If both properties are absent or empty, the ReadRestrictions object imposes no
+          restrictions and is equivalent to the field being absent from the response.
+          A server MUST NOT return an action for a column whose type is not listed in
+          that action's "Applicable to" set.
+          For all actions, if the input column value is NULL, the output MUST be NULL.
+
+          If a column projection targets a struct-typed field, other column projections
+          in the same ReadRestrictions MUST NOT target any of that struct's subfields
+          (at any depth). This avoids ambiguity about which action governs a given
+          leaf value.

Review Comment:
   Reposting my point as a reply here so the thread stays in one place — sorry for the parallel thread.
   
   > List<String> has 1 fieldID, map<String, int>, key have one fieldId and value would have another? i am unsure if the struct type restriction apply to them
   
   `list<string>` actually has two field IDs in the Iceberg schema model: the list field itself and the inner element. Same for `map`: one field ID for the map, then separate IDs for key and value. So a catalog could in principle return one action targeting `list<string>` (the container) and another action targeting the element's field ID, with no rule in the current spec saying which one wins for the leaf value.
   
   The struct-subfield rule on lines 3513–3516 already takes the position that ambiguity between a container and its descendants is bad. Suggest extending the same rule to all container kinds:
   
   > If a column projection targets field `F`, no other projection action in the same `ReadRestrictions` MAY target a field that is reachable from `F` by traversing struct subfields, list elements, or map keys/values (at any depth).
   
   That covers `list<struct<a, b>>` (action on the list + action on `a`), `map<string, int>` (action on the map + action on the value), and `list<string>` (action on the list + action on the element) with one rule.
   



##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3480,6 +3480,309 @@ components:
           additionalProperties:
             type: string
 
+    ReadRestrictions:
+      type: object
+      description: >
+          Read restrictions for a table, including column projections and row filter expressions.
+
+          A client MUST enforce the restrictions defined in this object when reading data
+          from the table.
+
+          These restrictions apply only to the authenticated principal, user, or account
+          associated with the request. They MUST NOT be interpreted as global policy and
+          MUST NOT be applied beyond the entity identified by the Authentication header
+          (or other applicable authentication mechanism).
+
+          If both properties are absent or empty, the ReadRestrictions object imposes no
+          restrictions and is equivalent to the field being absent from the response.
+          A server MUST NOT return an action for a column whose type is not listed in
+          that action's "Applicable to" set.
+          For all actions, if the input column value is NULL, the output MUST be NULL.
+
+          If a column projection targets a struct-typed field, other column projections
+          in the same ReadRestrictions MUST NOT target any of that struct's subfields
+          (at any depth). This avoids ambiguity about which action governs a given
+          leaf value.
+      properties:
+        required-column-projections:
+          description: >
+            A list of columns that require specific actions to be applied when reading.
+
+            If this property is absent, a reader MAY access all columns of the table as-is
+            without any mandatory transformations.
+
+            If this property is present, each listed column MUST have its specified
+            action applied. Columns not listed in required-column-projections
+            are not subject to any read restrictions.
+
+            When this list is present:
+
+            1. For each column listed in required-column-projections, the reader MUST apply
+              the specified action before returning values for that column.
+
+            2. The reader MUST replace all output references to the column with the result
+              of the action, presenting the result under the original column name. For
+              example, if the action for column cc is mask-alphanum, the reader MUST
+              return the masked value as cc in the query output.
+
+            3. Columns not listed in required-column-projections MAY be projected normally
+              by the reader without any mandatory transformations.
+
+            4. A column MUST appear at most once in required-column-projections.
+
+            5. If a projected column's action cannot be evaluated by the reader
+              (including unrecognized action types), the reader MUST fail rather than
+              ignore or skip the action.
+
+            6. Each action defines the output type for its column. For all predefined
+              actions except apply-expression, the output type matches the input column
+              type. For apply-expression, the output type is determined by the expression.
+
+          type: array
+          items:
+            $ref: '#/components/schemas/Action'
+        required-row-filter:
+          description: >
+            An expression that filters rows in the table that the authenticated principal does not have access to.
+
+            1. The expression MUST evaluate to a boolean. A reader MUST discard any row for which
+              the filter evaluates to FALSE, and no information derived from discarded rows
+              MAY be included in the query result.

Review Comment:
   Resolved by the new wording on line 3577–3580: `boolean (TRUE or FALSE; Iceberg expressions never produce NULL)`. Marking from my side. Thanks!
   



##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3480,6 +3480,309 @@ components:
           additionalProperties:
             type: string
 
+    ReadRestrictions:
+      type: object
+      description: >
+          Read restrictions for a table, including column projections and row filter expressions.
+
+          A client MUST enforce the restrictions defined in this object when reading data
+          from the table.
+
+          These restrictions apply only to the authenticated principal, user, or account
+          associated with the request. They MUST NOT be interpreted as global policy and
+          MUST NOT be applied beyond the entity identified by the Authentication header
+          (or other applicable authentication mechanism).
+
+          If both properties are absent or empty, the ReadRestrictions object imposes no
+          restrictions and is equivalent to the field being absent from the response.
+          A server MUST NOT return an action for a column whose type is not listed in
+          that action's "Applicable to" set.
+          For all actions, if the input column value is NULL, the output MUST be NULL.
+
+          If a column projection targets a struct-typed field, other column projections
+          in the same ReadRestrictions MUST NOT target any of that struct's subfields
+          (at any depth). This avoids ambiguity about which action governs a given
+          leaf value.
+      properties:
+        required-column-projections:
+          description: >
+            A list of columns that require specific actions to be applied when reading.
+
+            If this property is absent, a reader MAY access all columns of the table as-is
+            without any mandatory transformations.
+
+            If this property is present, each listed column MUST have its specified
+            action applied. Columns not listed in required-column-projections
+            are not subject to any read restrictions.
+
+            When this list is present:
+
+            1. For each column listed in required-column-projections, the reader MUST apply
+              the specified action before returning values for that column.
+
+            2. The reader MUST replace all output references to the column with the result
+              of the action, presenting the result under the original column name. For
+              example, if the action for column cc is mask-alphanum, the reader MUST
+              return the masked value as cc in the query output.
+
+            3. Columns not listed in required-column-projections MAY be projected normally
+              by the reader without any mandatory transformations.
+
+            4. A column MUST appear at most once in required-column-projections.
+
+            5. If a projected column's action cannot be evaluated by the reader
+              (including unrecognized action types), the reader MUST fail rather than
+              ignore or skip the action.
+
+            6. Each action defines the output type for its column. For all predefined
+              actions except apply-expression, the output type matches the input column
+              type. For apply-expression, the output type is determined by the expression.
+
+          type: array
+          items:
+            $ref: '#/components/schemas/Action'
+        required-row-filter:
+          description: >
+            An expression that filters rows in the table that the authenticated principal does not have access to.
+
+            1. The expression MUST evaluate to a boolean. A reader MUST discard any row for which
+              the filter evaluates to FALSE, and no information derived from discarded rows
+              MAY be included in the query result.
+
+            2. Row filters MUST be evaluated against the original, untransformed column values.
+              Required projections MUST be applied only after row filters are applied.
+
+            3. If a client cannot interpret or evaluate a provided filter expression, it MUST fail.
+
+            4. If this property is absent, null, or always true then no mandatory filtering is required.
+          $ref: '#/components/schemas/Expression'
+
+    Action:
+      discriminator:
+        propertyName: action
+        mapping:
+          mask-alphanum: '#/components/schemas/MaskAlphanum'
+          mask-to-fixed-value: '#/components/schemas/MaskToFixedValue'
+          replace-with-null: '#/components/schemas/ReplaceWithNull'
+          show-first-4: '#/components/schemas/ShowFirst4'
+          show-last-4: '#/components/schemas/ShowLast4'
+          truncate-to-year: '#/components/schemas/TruncateToYear'
+          truncate-to-month: '#/components/schemas/TruncateToMonth'
+          sha-256-global: '#/components/schemas/Sha256Global'
+          sha-256-query-local: '#/components/schemas/Sha256QueryLocal'
+          apply-expression: '#/components/schemas/ApplyExpression'
+      type: object
+      required:
+        - action
+        - field-id
+      properties:
+        action:
+          type: string
+        field-id:
+          type: integer
+          description: field id of the column being projected.
+
+    MaskAlphanum:
+      description: >
+        Redacts the column value Unicode code point by code point using the following rules:
+
+        - Digits (U+0030–U+0039, 0-9) are replaced with 'n'
+        - The following punctuation characters are kept as-is:
+            U+0028 '('  LEFT PARENTHESIS
+            U+0029 ')'  RIGHT PARENTHESIS
+            U+002C ','  COMMA
+            U+002E '.'  FULL STOP
+            U+002D '-'  HYPHEN-MINUS
+            U+0040 '@'  COMMERCIAL AT
+        - All other Unicode characters (including letters, whitespace, and any punctuation
+          not listed above) are replaced with 'x'
+
+        For example: "prashant010696@gmail.com" → "xxxxxxxxnnnnnn@xxxxx.xxx"
+
+        Applicable to: string
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      properties:
+        action:
+          type: string
+          const: "mask-alphanum"
+
+    MaskToFixedValue:
+      description: >
+        Replaces the column value with a predefined type-specific fixed value.
+        Engines MUST use exactly the values listed below to ensure consistency
+        across implementations.
+
+        Fixed values by type:
+        - boolean: false
+        - int: 0
+        - long: 0
+        - float: 0.0
+        - double: 0.0
+        - decimal(p, s): 0 (zero with s digits after the decimal point, e.g. 0.00 for decimal(p,2))
+        - string: "XXXXXXXX"
+        - date: 1970-01-01
+        - time: 00:00:00
+        - timestamp: 1970-01-01T00:00:00
+        - timestamptz: 1970-01-01T00:00:00+00:00
+        - timestamp_ns: 1970-01-01T00:00:00.000000000
+        - timestamptz_ns: 1970-01-01T00:00:00.000000000+00:00
+        - uuid: 00000000-0000-0000-0000-000000000000
+        - fixed(n): n zero bytes
+        - binary: empty byte sequence
+        - variant: {}
+        - geometry: POINT EMPTY
+        - geography: POINT EMPTY
+        - list: empty list []
+        - map: empty map {}
+        - struct: struct with each field set to its type-specific default (applied recursively)
+
+        Applicable to: all data types
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      properties:
+        action:
+          type: string
+          const: "mask-to-fixed-value"
+
+    ReplaceWithNull:
+      description: >
+        Replaces the entire column value with NULL.
+
+        Applicable to: all nullable types
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      properties:
+        action:
+          type: string
+          const: "replace-with-null"
+
+    ShowFirst4:
+      description: >
+        Preserves the first 4 Unicode code points of the column value and redacts the remainder
+        using mask-alphanum rules (see MaskAlphanum for the exact character rules).
+        Values with 4 or fewer Unicode code points are returned unchanged.
+
+        For example: "prashant010696@gmail.com" → "prasxxxxnnnnnn@xxxxx.xxx"
+
+        Applicable to: string
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      properties:
+        action:
+          type: string
+          const: "show-first-4"
+
+    ShowLast4:
+      description: >
+        Redacts all Unicode code points except the last 4 using mask-alphanum rules
+        (see MaskAlphanum for the exact character rules).
+        Values with 4 or fewer Unicode code points are returned unchanged.
+
+        For example: "4111-1111-1111-4444" → "nnnn-nnnn-nnnn-4444"
+
+        Applicable to: string
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      properties:
+        action:
+          type: string
+          const: "show-last-4"
+
+    TruncateToYear:
+      description: >
+        Truncates the column value to year precision, setting month, day, and time components
+        to their minimum values. The output type matches the input type.
+
+        For example: 2024-07-15 → 2024-01-01
+        For timestamptz and timestamptz_ns, truncation is performed in UTC.
+
+        Applicable to: date, timestamp, timestamptz, timestamp_ns, timestamptz_ns
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      properties:
+        action:
+          type: string
+          const: "truncate-to-year"
+
+    TruncateToMonth:
+      description: >
+        Truncates the column value to year and month precision, setting day and time components
+        to their minimum values. The output type matches the input type.
+
+        For example: 2024-07-15 → 2024-07-01
+        For timestamptz and timestamptz_ns, truncation is performed in UTC.
+
+        Applicable to: date, timestamp, timestamptz, timestamp_ns, timestamptz_ns
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      properties:
+        action:
+          type: string
+          const: "truncate-to-month"
+
+    Sha256Global:
+      description: |
+        Applies SHA-256 as specified in NIST FIPS 180-4. Deterministic across all queries
+        and engines — the same input always produces the same output.
+
+        Input-to-bytes encoding by type:
+        - string: UTF-8 encoded bytes
+        - int: 4 bytes, little-endian
+        - long: 8 bytes, little-endian
+        - binary: raw bytes as-is
+
+        Output encoding by type:
+        - string: 64-character lowercase hexadecimal string
+        - int: first 4 bytes of the digest, read as a signed two's complement little-endian int
+        - long: first 8 bytes of the digest, read as a signed two's complement little-endian long
+        - binary: the full 32-byte raw SHA-256 digest
+
+        Applicable to: string, int, long, binary
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      properties:
+        action:
+          type: string
+          const: "sha-256-global"
+
+    Sha256QueryLocal:
+      description: |
+        Applies SHA-256 with a per-query random salt, making the output non-deterministic
+        across queries while remaining consistent within a single query.
+
+        The engine MUST generate a cryptographically random salt of at least 16 bytes for each query and apply it as:
+          SHA-256(salt_bytes || canonical_bytes)
+        where canonical_bytes follows the same encoding rules as sha-256-global.
+
+        Output encoding follows the same rules as sha-256-global.
+
+        Applicable to: string, int, long, binary
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      properties:
+        action:
+          type: string
+          const: "sha-256-query-local"
+
+    ApplyExpression:
+      description: >
+        Replace the field with the result of an expression. Produce the original field name
+        with the expression result.
+
+        Applicable to: all data types
+      allOf:
+        - $ref: '#/components/schemas/Action'
+      required:
+        - action
+        - expression
+      properties:
+        action:
+          type: string
+          const: "apply-expression"
+        expression:
+          $ref: '#/components/schemas/Expression'

Review Comment:
   Moot: `apply-expression` was removed in commit 9055de87. Closing from my side.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org