You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/01/24 02:01:31 UTC
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
RussellSpitzer commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r790363699
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
Do we have the possibility of CommitStateUnknown exceptions here as well?
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
Review comment:
I think for all the "add" commands we should add that to the description, ("Does not change the current Spec/Schema/Snapshot")
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
Review comment:
Are we defining this as always setting the current schema to the added schema? I think it's fine not too since we have the "set current schema" step below but I think we need to document that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org