You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/01/23 22:31:03 UTC
[GitHub] [iceberg] rdblue opened a new pull request #3955: Spec: Add REST basics for TableOperations
rdblue opened a new pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955
This adds two new routes to the REST catalog spec:
* `GET /v1/namespaces/{namespace}/tables/{table}` is used to load a table. This returns:
* Table-specific configuration for the catalog (e.g., a specific FileIO implementation for the table)
* The current table metadata
* `POST /v1/namespaces/{namespace}/tables/{table}` is used to commit updates a table
* The request body contains `requirements` and `updates` as in the [REST commit proposal](https://docs.google.com/document/d/1D0R3G0slssEhggH5XnIzMwsUIP-c385Qp2sjv5E7e6E/edit#heading=h.eo4x0coo8esy), but complete
* The response contains the updated table metadata
This PR includes the metadata JSON format expressed in OpenAPI to be able to return full `TableMetadata`.
Note that this updates the proposed change set slightly to account for the addition of branching and tagging. Instead of `SetCurrentSnapshot` the change is `SetSnapshotRef`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791965836
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
Review comment:
We might want to include `403` status code as well. `401` pertains to authentication where `403` pertains to authorization. It's worth distinguishing between the two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794081522
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
+
+ SetCurrentSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema-id
+ properties:
+ schema-id:
+ type: integer
+ description: Schema ID to set as current, or -1 to set last added schema
+
+ AddPartitionSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec
+ properties:
+ spec:
+ $ref: '#/components/schemas/PartitionSpec'
+
+ SetDefaultSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec-id
+ properties:
+ spec-id:
+ type: integer
+ description: Partition spec ID to set as the default, or -1 to set last added spec
+
+ AddSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - sort-order
+ properties:
+ sort-order:
+ $ref: '#/components/schemas/SortOrder'
+
+ SetDefaultSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - order-id
+ properties:
+ order-id:
+ type: integer
+ description: Sort order ID to set as the default, or -1 to set last added sort order
+
+ AddSnapshotUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - snapshot
+ properties:
+ snapshot:
+ $ref: '#/components/schemas/Snapshot'
+
+ SetSnapshotRefUpdate:
Review comment:
Yes, this is updated for the tagging and branching work. Tags and branches are tracked by refs, which are named snapshots. The "current" table state is held by the "main" branch. So `SetCurrentSnapshot` would translate to this:
```json
{
"action": "set-snapshot-ref",
"ref": "main",
"snaphot-id": 65987125071833
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r790583606
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
- description: Reference to one or more levels of a namespace
- type: array
- items:
- type: string
- example: [ "accounting", "tax" ]
+ UpdateNamespacePropertiesRequest:
Review comment:
It's kind of weird there's not a dedicated `requests` section, huh?
Eventually, if we wanted to, we could place things into separate files. But then we'd need something to bundle them all up together so the trade-off might not be worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794077250
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
- description: Reference to one or more levels of a namespace
- type: array
- items:
- type: string
- example: [ "accounting", "tax" ]
+ UpdateNamespacePropertiesRequest:
+ type: object
+ properties:
+ removals:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
Review comment:
Good question for @kbendick!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794079325
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
Review comment:
This is to match the Iceberg spec, where the refs are a map of ref name to ref data. We could also have a ref that embeds its name and keep them in a list. That's what we do for schemas, partition specs, etc. Either way, what we do here should match the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791418778
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
Review comment:
Yeah this could be changed to simply `TableToLoadDoesNotExist`. The key here in the examples map doesn’t have many requirements on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r790339964
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
- description: Reference to one or more levels of a namespace
- type: array
- items:
- type: string
- example: [ "accounting", "tax" ]
+ UpdateNamespacePropertiesRequest:
Review comment:
Moved this from elsewhere. Everything else is additions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r792900276
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -814,6 +1423,34 @@ components:
Server's do not need to implement this.
nullable: true
+ LoadTableResponse:
+ description: Table metadata result when loading a table
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - metadata
+ properties:
+ config:
+ type: object
+ additionalProperties:
+ type: string
+ metadata:
+ $ref: '#/components/schemas/TableMetadata'
+
+ CommitTableResponse:
Review comment:
Right now this is just focused on normal table updates. I think we need to layer on CREATE TABLE and transactional semantics in a separate PR. I think it will probably look like adding an option to the commit and the create, but I've not gotten it completely worked out yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794948276
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +719,588 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
Review comment:
Nit: Can we put a `min` and `max` here? I used the `widdershins` CLI tool to generate markdown, and the example `format-version` it gave was 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794947539
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
- description: Reference to one or more levels of a namespace
- type: array
- items:
- type: string
- example: [ "accounting", "tax" ]
+ UpdateNamespacePropertiesRequest:
+ type: object
+ properties:
+ removals:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
Review comment:
Actually, another CLI tool I put it in called `widdershins` didn't like it. They should be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#issuecomment-1024774708
Thanks for the reviews, @RussellSpitzer, @danielcweeks, @jackye1995, @kbendick, @szehon-ho, and @bryanck!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794874647
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
Review comment:
```
Primitive data types in the OAS are based on the types supported by the JSON Schema Specification Wright Draft 00.
Note that integer as a type is also supported and is defined as a JSON number without a fraction or exponent part. null is not supported as a type (see nullable for an alternative solution).
Models are defined using the Schema Object, which is an extended subset of JSON Schema Specification Wright Draft 00.
Primitives have an optional modifier property: format. OAS uses several known formats to define in fine detail the data type being used.
However, to support documentation needs, the format property is an open string-valued property, and can have any value. Formats such as "email", "uuid", and so on, MAY be used even though undefined by this specification.
Types that are not accompanied by a format property follow the type definition in the JSON Schema.
Tools that do not recognize a specific format MAY default back to the type alone, as if the format is not specified.
The formats defined by the OAS are:
type | format | Comments
-- | -- | --
integer | int32 | signed 32 bits
integer | int64 | signed 64 bits (a.k.a long)
...
```
Not sure it matters to JSON as @rdblue said, but couldn't hurt 🤷
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791246200
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
Good question. I think that we do have the possibility of unknown commit state for the same reasons we have it in the other catalog clients. If this is sending the update off to a DB and gets disconnected, then there's a possibility that the commit succeeded but the service doesn't know about it. We can't guarantee that the service endpoint always knows the final commit state.
However, I don't think that this should actually send `CommitStateUnknown`. Instead, it should send a 5xx error and trigger commit state recovery on the client side. That way, we don't have retries within retries. The client would get the complete server-side error, then query the table to see whether the commit succeeded. If it did, then it would return success and if not would throw its own `CommitStateUnknownException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794098319
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
I updated this so that the description documents what to do. After looking through the list of [status codes](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes), I think that only 500 (generic error) and 504 (gateway timeout) should leave the table in an unknown state. Other error codes indicate that the operation wasn't attempted, like 501 (not implemented).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r792310852
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
> Instead, it should send a 5xx error and trigger commit state recovery on the client side.
Yes a 5XX sounds like the right way for commit state unknown. This is probably a good one to distinguish 503 and 500, where 500 represents the service is likely down and commit state is unknown, 503 means the user is throttled maybe because of too many requests, but we are sure the commit failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r792902149
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
Review comment:
Should it be array type? (I couldn't see easily what this corresponds to in the code)
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
Review comment:
Just checking, you are intentionally dropping the 'newLastColumnId' argument in the TableMetadata.Builder equivalent for simplicity right?
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
Review comment:
Should be array of SortField , right?
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
+
+ SetCurrentSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema-id
+ properties:
+ schema-id:
+ type: integer
+ description: Schema ID to set as current, or -1 to set last added schema
+
+ AddPartitionSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec
+ properties:
+ spec:
+ $ref: '#/components/schemas/PartitionSpec'
+
+ SetDefaultSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec-id
+ properties:
+ spec-id:
+ type: integer
+ description: Partition spec ID to set as the default, or -1 to set last added spec
+
+ AddSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - sort-order
+ properties:
+ sort-order:
+ $ref: '#/components/schemas/SortOrder'
+
+ SetDefaultSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - order-id
+ properties:
+ order-id:
+ type: integer
+ description: Sort order ID to set as the default, or -1 to set last added sort order
+
+ AddSnapshotUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - snapshot
+ properties:
+ snapshot:
+ $ref: '#/components/schemas/Snapshot'
+
+ SetSnapshotRefUpdate:
Review comment:
Is it replacing 'setCurrentSnapshot' in the code? Just for my understanding, what does 'ref' represent?
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
- description: Reference to one or more levels of a namespace
- type: array
- items:
- type: string
- example: [ "accounting", "tax" ]
+ UpdateNamespacePropertiesRequest:
+ type: object
+ properties:
+ removals:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
Review comment:
Are there extra single quotes ? (I see its moved from outside and not related to this change)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794973272
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
Review comment:
https://github.com/apache/iceberg/issues/4001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794937450
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
- description: Reference to one or more levels of a namespace
- type: array
- items:
- type: string
- example: [ "accounting", "tax" ]
+ UpdateNamespacePropertiesRequest:
+ type: object
+ properties:
+ removals:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
Review comment:
The extra single quotes are allowed from my experience. It will simply parse that whole thing as a JSON string (at least the various UI editors I've tried have allowed that).
That said, I would remove them for consistency with the other examples.
Here's the output from editor.swagger.io, where it is parsed correctly:
```
example: [ "department", "access_group" ]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794970600
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
Review comment:
Yes, please do!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794081522
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
+
+ SetCurrentSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema-id
+ properties:
+ schema-id:
+ type: integer
+ description: Schema ID to set as current, or -1 to set last added schema
+
+ AddPartitionSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec
+ properties:
+ spec:
+ $ref: '#/components/schemas/PartitionSpec'
+
+ SetDefaultSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec-id
+ properties:
+ spec-id:
+ type: integer
+ description: Partition spec ID to set as the default, or -1 to set last added spec
+
+ AddSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - sort-order
+ properties:
+ sort-order:
+ $ref: '#/components/schemas/SortOrder'
+
+ SetDefaultSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - order-id
+ properties:
+ order-id:
+ type: integer
+ description: Sort order ID to set as the default, or -1 to set last added sort order
+
+ AddSnapshotUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - snapshot
+ properties:
+ snapshot:
+ $ref: '#/components/schemas/Snapshot'
+
+ SetSnapshotRefUpdate:
Review comment:
Yes, this is updated for the tagging and branching work. Tags and branches are tracked by refs, which are named snapshots. The "current" table state is held by the "main" branch. So `SetCurrentSnapshot` would translate to this:
```json
{
"action": "set-snapshot-ref",
"ref": "main",
"snapshot-id": 65987125071833
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r790340043
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
+ type: string
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
+
+ SetCurrentSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema-id
+ properties:
+ schema-id:
+ type: integer
+ description: Schema ID to set as current, or -1 to set last added schema
+
+ AddPartitionSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec
+ properties:
+ spec:
+ $ref: '#/components/schemas/PartitionSpec'
+
+ SetDefaultSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec-id
+ properties:
+ spec-id:
+ type: integer
+ description: Partition spec ID to set as the default, or -1 to set last added spec
+
+ AddSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - sort-order
+ properties:
+ sort-order:
+ $ref: '#/components/schemas/SortOrder'
+
+ SetDefaultSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - order-id
+ properties:
+ order-id:
+ type: integer
+ description: Sort order ID to set as the default, or -1 to set last added sort order
+
+ AddSnapshotUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - snapshot
+ properties:
+ snapshot:
+ $ref: '#/components/schemas/Snapshot'
+
+ SetSnapshotRefUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - $ref: '#/components/schemas/SnapshotReference'
+ - type: object
+ required:
+ - ref
+ properties:
+ ref:
+ type: string
+
+ RemoveSnapshotsUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - snapshot-ids
+ properties:
+ snapshot-ids:
+ type: array
+ items:
+ type: integer
+
+ SetLocationUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - location
+ properties:
+ location:
+ type: string
+
+ SetPropertiesUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - updates
+ properties:
+ updates:
+ type: object
+ additionalProperties:
+ type: string
+
+ RemovePropertiesUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - removals
+ properties:
+ removals:
+ type: array
+ items:
+ type: string
+
+ TableUpdate:
Review comment:
These are all of the [updates proposed in the design doc](https://docs.google.com/document/d/1D0R3G0slssEhggH5XnIzMwsUIP-c385Qp2sjv5E7e6E/edit#heading=h.eo4x0coo8esy).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r790340164
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
+ type: string
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
Review comment:
`TableMetadata` comes directly from the [spec](https://iceberg.apache.org/spec/#appendix-c-json-serialization) and [reference implementation parsers](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/TableMetadataParser.java).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791978008
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
Review comment:
This sounds like a good change as a follow-up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791416437
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
In the previous PR, we put `50X` everywhere. I believe I also included a section that mentions 50X can return text/plain (eg some middleware or proxy server hit an issue and then returned an HTML page, which definitely happens even once you hit the app sever).
I think we should include them everywhere, but I can understand possibly adding them last as they’re the same throughout,
50X response would be maybe a good argument for using an external tool to bundle all of the files together, though we’re likely far from being in that position currently and should focus on defining the spec first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794081522
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
+
+ SetCurrentSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema-id
+ properties:
+ schema-id:
+ type: integer
+ description: Schema ID to set as current, or -1 to set last added schema
+
+ AddPartitionSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec
+ properties:
+ spec:
+ $ref: '#/components/schemas/PartitionSpec'
+
+ SetDefaultSpecUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - spec-id
+ properties:
+ spec-id:
+ type: integer
+ description: Partition spec ID to set as the default, or -1 to set last added spec
+
+ AddSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - sort-order
+ properties:
+ sort-order:
+ $ref: '#/components/schemas/SortOrder'
+
+ SetDefaultSortOrderUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - order-id
+ properties:
+ order-id:
+ type: integer
+ description: Sort order ID to set as the default, or -1 to set last added sort order
+
+ AddSnapshotUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - snapshot
+ properties:
+ snapshot:
+ $ref: '#/components/schemas/Snapshot'
+
+ SetSnapshotRefUpdate:
Review comment:
Yes, this is updated for the tagging and branching work. Tags and branches are tracked by refs, which are named snapshots. The "current" table state is held by the "main" branch. So `SetCurrentSnapshot` would translate to this:
```json
{
"action": "set-snapshot-ref",
"ref": "main",
"snaphot-id": 65987125071833
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794970722
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
Review comment:
I'm find adding format annotations, but we should consider doing that everywhere, which is pretty far out of scope for this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r792201260
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
Review comment:
I think that anything can be done using the order of updates. If I have two add schema updates, I can either put the set schema after the first or after the second.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791244069
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
Review comment:
No. None of the adds actually change the schema. That's why I have the "set" updates. This handles ID assignment by using `-1` as a signal value to set the current schema to the one that was just added.
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
Good question. I think that we do have the possibility of unknown commit state for the same reasons we have it in the other catalog clients. If this is sending the update off to a DB and gets disconnected, then there's a possibility that the commit succeeded but the service doesn't know about it. We can't guarantee that the service endpoint always knows the final commit state.
However, I don't think that this should actually send `CommitStateUnknown`. Instead, it should send a 5xx error and trigger commit state recovery on the client side. That way, we don't have retries within retries. The client would get the complete server-side error, then query the table to see whether the commit succeeded. If it did, then it would return success and if not would throw its own `CommitStateUnknownException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r792311446
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
with that being said, it seems like we should always distinguish 500 and 503? Just realized what I said is probably true for all the other APIs, but this one we want to return a different Java exception to users based on the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r790358987
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
Review comment:
Looks like copy/paste error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791417463
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
Review comment:
Do we want to do this in another PR to reduce clutter? I agree they make sense together, as there’s no section for `requests`.
I’d alphabetize actual schema elements and then separate them by a block that does requests. I can open a PR for that if you’d like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
bryanck commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r792878396
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -814,6 +1423,34 @@ components:
Server's do not need to implement this.
nullable: true
+ LoadTableResponse:
+ description: Table metadata result when loading a table
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - metadata
+ properties:
+ config:
+ type: object
+ additionalProperties:
+ type: string
+ metadata:
+ $ref: '#/components/schemas/TableMetadata'
+
+ CommitTableResponse:
Review comment:
Do we want to specify the transaction type someplace, e.g. simple, create_table, replace_table, etc.? Alternatively that could go in a token claim or something similar. For example, if there are two create table (without replace) transactions started at the same time, then the first one commits. When the second one commits, a catalog may want to prevent the replace from happening if it wasn't specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794936262
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -362,11 +362,142 @@ paths:
5XX:
$ref: '#/components/responses/ServerErrorResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table in the given namespace
+ description: Create table
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '#/components/examples/NoSuchNamespaceError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ /v1/namespaces/{namespace}/tables:
Review comment:
Bug: I think this line is intended to be above the `post` section above? I believe it can just be removed.
When loading this into `editor.swagger.io`, I get the following error:
```
Parser error
duplicated mapping key
Jump to line 395
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794941878
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
Review comment:
Yeah I'm in agreement that it doesn't matter to the JSON, but if somebody were to do code generation, it might be good to have in cases where it's explicitly defined in our spec.
I'm fine either way as it might add more overhead in reading through it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794948276
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +719,588 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
Review comment:
Nit: Can we put a `minimum` and `maximum` here? I used the `widdershins` CLI tool to generate markdown, and the example `format-version` it gave was 0.
At the least, `minimum` of 1 should be put.
```
properties:
format-version:
type: integer
minimum: 1
maximum: 2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r790339936
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
Review comment:
Moved this near `TableIdentifier`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791425138
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
Review comment:
The integer width is defined in the spec, right? We should define if here too. I believe the key is `format` and can be `int32` or `int64` (eg long).
You can also use type Number and then use `float` and `double`. `Number` is the super type basically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791416437
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
In the previous PR, we put `50X` everywhere. I believe I also included a section that mentions 50X can return text/plain (eg some middleware or proxy server hit an issue and then returned an HTML page, which definitely happens even once you hit the app sever).
I think we should include them everywhere, but I can understand possibly adding them last as they’re the same throughout,
50X response would be maybe a good argument for using an external tool to bundle all of the files together, though we’re likely far from being in that position currently and should focus on defining the spec first.
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
Review comment:
Do we want to do this in another PR to reduce clutter? I agree they make sense together, as there’s no section for `requests`.
I’d alphabetize actual schema elements and then separate them by a block that does requests. I can open a PR for that if you’d like.
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
Review comment:
Yeah this could be changed to simply `TableToLoadDoesNotExist`. The key here in the examples map doesn’t have many requirements on it.
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
In the previous PR, we put `50X` everywhere. I believe I also included a section that mentions 50X can return text/plain (eg some middleware or proxy server hit an issue and then returned an HTML page, which definitely happens even once you hit the app sever).
I think we should include them everywhere, but I can understand possibly adding them last as they’re the same throughout,
50X response would be maybe a good argument for using an external tool to bundle all of the files together, though we’re likely far from being in that position currently and should focus on defining the spec first.
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
Review comment:
+1. This is the proper way to show generic string to string map (though it looks weird if you’re not used to it).
I’ve considered defining a `StringMap` type as I’ve seen that done in some projects to make the `additionalProperties` part less weird (especially for `Map<String, String>` where we know the required fields and other things.
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
Review comment:
The integer width is defined in the spec, right? We should define if here too. I believe the key is `format` and can be `int32` or `int64` (eg long).
You can also use type Number and then use `float` and `double`. `Number` is the super type basically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794080423
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
Review comment:
Yes. That can always be recovered as `max(currentLastColumnId, max(columnIdsFromNewSchema))`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794936262
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -362,11 +362,142 @@ paths:
5XX:
$ref: '#/components/responses/ServerErrorResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table in the given namespace
+ description: Create table
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '#/components/examples/NoSuchNamespaceError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ /v1/namespaces/{namespace}/tables:
Review comment:
Possible bug: I think this line is intended to be above the `post` section above? it seems off to have two routes, one after another.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794937450
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
- description: Reference to one or more levels of a namespace
- type: array
- items:
- type: string
- example: [ "accounting", "tax" ]
+ UpdateNamespacePropertiesRequest:
+ type: object
+ properties:
+ removals:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
Review comment:
The extra single quotes are allowed from my experience. It will simply parse that whole thing as a JSON string (at least the various UI editors I've tried have allowed that).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791276619
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
So we should probably put a note in 50X Server Error, Operation may or may not have succeeded? Recheck before retrying? I just feel like we need to specify it so implementers know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791244069
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
Review comment:
No. None of the adds actually change the schema. That's why I have the "set" updates. This handles ID assignment by using `-1` as a signal value to set the current schema to the one that was just added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791978449
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
Review comment:
I don't think that JSON distinguishes between longs and integers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791979725
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
Review comment:
I agree with this. We should add 403 to all of the routes in a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791416437
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
In the previous PR, we put `50X` everywhere. I believe I also included a section that mentions 50X can return text/plain (eg some middleware or proxy server hit an issue and then returned an HTML page, which definitely happens even once you hit the app sever).
I think we should include them everywhere, but I can understand possibly adding them last as they’re the same throughout,
50X response would be maybe a good argument for using an external tool to bundle all of the files together, though we’re likely far from being in that position currently and should focus on defining the spec first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794076822
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
Review comment:
Let's leave it as is for now. We'll make minor changes over time, I'm sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794936503
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
Review comment:
I can handle 403 in a follow up PR if we'd like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794973588
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
Review comment:
https://github.com/apache/iceberg/issues/4002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794092703
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
Review comment:
Yep. Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791982585
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
Review comment:
Do we need to distinguish between multiple schema updates? Or is the order implied in the request? For example if I pass in two AddSchemaUpdates, which is last (-1)? Maybe this is an edge case and we don't really need to worry about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791276619
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
So we should probably put a note in 50X Server Error, Operation may or may not have succeeded? Recheck before retrying? I just feel like we need to specify it so implementers know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r792309269
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
Review comment:
+1 for adding 403 for all routes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r791421444
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
Review comment:
+1. This is the proper way to show generic string to string map (though it looks weird if you’re not used to it).
I’ve considered defining a `StringMap` type as I’ve seen that done in some projects to make the `additionalProperties` part less weird (especially for `Map<String, String>` where we know the required fields and other things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794937450
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
- description: Reference to one or more levels of a namespace
- type: array
- items:
- type: string
- example: [ "accounting", "tax" ]
+ UpdateNamespacePropertiesRequest:
+ type: object
+ properties:
+ removals:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
Review comment:
The extra single quotes are allowed from my experience. It will simply parse that whole thing as a JSON string (at least the various UI editors I've tried have allowed that).
That said, I would remove them for consistency with the other examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794973395
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -362,11 +362,142 @@ paths:
5XX:
$ref: '#/components/responses/ServerErrorResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table in the given namespace
+ description: Create table
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '#/components/examples/NoSuchNamespaceError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ /v1/namespaces/{namespace}/tables:
Review comment:
Fixed.
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +719,588 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r794077081
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -564,12 +648,21 @@ components:
example: '{ "owner": "Hank Bendickson" }'
default: '{ }'
- Namespace:
- description: Reference to one or more levels of a namespace
- type: array
- items:
- type: string
- example: [ "accounting", "tax" ]
+ UpdateNamespacePropertiesRequest:
Review comment:
You can have a section of `requestBodies`. I considered refactoring to use that, but it is a larger change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3955: Spec: Add REST basics for TableOperations
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3955:
URL: https://github.com/apache/iceberg/pull/3955#discussion_r790363699
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -367,6 +367,90 @@ paths:
- $ref: '#/components/parameters/namespace'
- $ref: '#/components/parameters/table'
+ get:
+ tags:
+ - Catalog API
+ summary: Load a table from the catalog
+ operationId: loadTable
+ description:
+ Load a table from the catalog.
+
+ The response contains both configuration and table metadata. The configuration, if non-empty is used
+ as additional configuration for the table that overrides catalog configuration. For example, this
+ configuration may change the FileIO implemented used for the table.
+
+ The response also contains the table's full metadata.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 5XX:
+ $ref: '#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Catalog API
+ summary: Commit updates to a table
+ operationId: updateTable
+ description:
+ Commit updates to a table.
+
+ Commits have two parts, requirements and updates. Requirements are assertions that will be validated
+ before attempting to make and commit changes. For example, assert-ref-snapshot-id will check that a
+ named ref's snapshot ID has a certain value.
+
+ Updates are changes to make to table metadata. For example, after asserting that the current main ref
+ is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new
+ snapshot id.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ responses:
+ 200:
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/CommitTableResponse'
+ 400:
+ $ref: '#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '#/components/responses/UnauthorizedResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableException, table to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergErrorResponse'
+ examples:
+ TableToRenameDoesNotExist:
+ $ref: '#/components/examples/NoSuchTableError'
+ 409:
+ description:
+ Conflict - CommitFailedException, one or more requirements failed. The client may retry.
Review comment:
Do we have the possibility of CommitStateUnknown exceptions here as well?
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
Review comment:
I think for all the "add" commands we should add that to the description, ("Does not change the current Spec/Schema/Snapshot")
##########
File path: rest_docs/rest-catalog-open-api.yaml
##########
@@ -579,36 +672,552 @@ components:
destination:
$ref: '#/components/schemas/TableIdentifier'
+ Namespace:
+ description: Reference to one or more levels of a namespace
+ type: array
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+
TableIdentifier:
type: object
required:
- namespace
- name
properties:
namespace:
+ $ref: '#/components/schemas/Namespace'
+ name:
+ type: string
+ nullable: false
+
+ PrimitiveType:
+ type: string
+ example:
+ - "long"
+ - "string"
+ - "fixed[16]"
+ - "decimal(10,2)"
+
+ StructField:
+ type: object
+ required:
+ - id
+ - name
+ - required
+ - type
+ properties:
+ id:
+ type: integer
+ name:
+ type: string
+ required:
+ type: boolean
+ type:
+ $ref: '#/components/schemas/Type'
+ doc:
+ type: string
+
+ StructType:
+ type: object
+ required:
+ - fields
+ properties:
+ type:
+ type: string
+ enum: ["struct"]
+ fields:
type: array
- description: Individual levels of the namespace
items:
- type: string
+ $ref: '#/components/schemas/StructField'
+
+ ListType:
+ type: object
+ required:
+ - type
+ - element-id
+ - element-required
+ - element
+ properties:
+ type:
+ type: string
+ enum: ["list"]
+ element-id:
+ type: integer
+ element-required:
+ type: boolean
+ element:
+ $ref: '#/components/schemas/Type'
+
+ MapType:
+ type: object
+ required:
+ - type
+ - key-id
+ - key
+ - value-id
+ - value-required
+ - value
+ properties:
+ type:
+ type: string
+ enum: ["map"]
+ key-id:
+ type: integer
+ key:
+ $ref: '#/components/schemas/Type'
+ value-id:
+ type: integer
+ value-required:
+ type: boolean
+ value:
+ $ref: '#/components/schemas/Type'
+
+ Type:
+ anyOf:
+ - $ref: '#/components/schemas/PrimitiveType'
+ - $ref: '#/components/schemas/StructType'
+ - $ref: '#/components/schemas/ListType'
+ - $ref: '#/components/schemas/MapType'
+
+ Schema:
+ allOf:
+ - $ref: '#/components/schemas/StructType'
+ - type: object
+ properties:
+ schema-id:
+ type: integer
+ readOnly: true
+ identifier-field-ids:
+ type: array
+ items:
+ type: integer
+
+ Transform:
+ type: string
+ example:
+ - "identity"
+ - "year"
+ - "month"
+ - "day"
+ - "hour"
+ - "bucket[256]"
+ - "truncate[16]"
+
+ PartitionField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - name
+ properties:
+ field-id:
+ type: integer
+ source-id:
+ type: integer
name:
type: string
- nullable: false
+ transform:
+ $ref: '#/components/schemas/Transform'
- UpdateNamespacePropertiesRequest:
+ PartitionSpec:
type: object
+ required:
+ - fields
properties:
- removals:
+ spec-id:
+ type: integer
+ readOnly: true
+ fields:
type: array
- uniqueItems: true
items:
+ $ref: '#/components/schemas/PartitionField'
+
+ SortDirection:
+ type: string
+ enum: ["asc", "desc"]
+
+ NullOrder:
+ type: string
+ enum: ["nulls-first", "nulls-last"]
+
+ SortField:
+ type: object
+ required:
+ - source-id
+ - transform
+ - direction
+ - null-order
+ properties:
+ source-id:
+ type: integer
+ transform:
+ $ref: '#/components/schemas/Transform'
+ direction:
+ $ref: '#/components/schemas/SortDirection'
+ null-order:
+ $ref: '#/components/schemas/NullOrder'
+
+ SortOrder:
+ type: object
+ required:
+ - fields
+ properties:
+ order-id:
+ type: integer
+ readOnly: true
+ fields:
+ $ref: '#/components/schemas/SortField'
+
+ Snapshot:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ - manifest-list
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+ manifest-list:
+ type: string
+ description: Location of the snapshot's manifest list file
+ schema-id:
+ type: integer
+ summary:
+ type: object
+ required:
+ - summary
+ properties:
+ summary:
+ type: string
+ enum: ["append", "replace", "overwrite", "delete"]
+ additionalProperties:
+ type: string
+
+ SnapshotReference:
+ type: object
+ required:
+ - type
+ - snapshot-id
+ properties:
+ type:
+ type: string
+ enum: ["tag", "branch"]
+ snapshot-id:
+ type: integer
+ max-ref-age-ms:
+ type: integer
+ max-snapshot-age-ms:
+ type: integer
+ min-snapshots-to-keep:
+ type: integer
+
+ SnapshotReferences:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/SnapshotReference'
+
+ SnapshotLog:
+ type: array
+ items:
+ type: object
+ required:
+ - snapshot-id
+ - timestamp-ms
+ properties:
+ snapshot-id:
+ type: integer
+ timestamp-ms:
+ type: integer
+
+ MetadataLog:
+ type: array
+ items:
+ type: object
+ required:
+ - metadata-file
+ - timestamp-ms
+ properties:
+ metadata-file:
type: string
- example: '[ "department", "access_group" ]'
- updates:
- uniqueItems: true
+ timestamp-ms:
+ type: integer
+
+ TableMetadata:
+ type: object
+ required:
+ - format-version
+ - table-uuid
+ properties:
+ format-version:
+ type: integer
+ table-uuid:
+ type: string
+ location:
+ type: string
+ last-updated-ms:
+ type: integer
+ properties:
type: object
- items:
+ additionalProperties:
type: string
- example: { "owner": "Hank Bendickson" }
+ # schema tracking
+ schemas:
+ type: array
+ items:
+ $ref: '#/components/schemas/Schema'
+ current-schema-id:
+ type: integer
+ last-column-id:
+ type: integer
+ # partition spec tracking
+ partition-specs:
+ type: array
+ items:
+ $ref: '#/components/schemas/PartitionSpec'
+ default-spec-id:
+ type: integer
+ last-partition-id:
+ type: integer
+ # sort order tracking
+ sort-orders:
+ type: array
+ items:
+ $ref: '#/components/schemas/SortOrder'
+ default-sort-order-id:
+ type: integer
+ # snapshot tracking
+ snapshots:
+ type: array
+ items:
+ $ref: '#/components/schemas/Snapshot'
+ refs:
+ $ref: '#/components/schemas/SnapshotReferences'
+ current-snapshot-id:
+ type: integer
+ # logs
+ snapshot-log:
+ $ref: '#/components/schemas/SnapshotLog'
+ metadata-log:
+ $ref: '#/components/schemas/MetadataLog'
+
+ BaseUpdate:
+ type: object
+ required:
+ - action
+ properties:
+ action:
+ type: string
+ enum:
+ - upgrade-format-version
+ - add-schema
+ - set-current-schema
+ - add-spec
+ - set-default-spec
+ - add-sort-order
+ - set-default-sort-order
+ - add-snapshot
+ - set-snapshot-ref
+ - remove-snapshots
+ - set-location
+ - set-properties
+ - remove-properties
+
+ UpgradeFormatVersionUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - format-version
+ properties:
+ format-version:
+ type: integer
+
+ AddSchemaUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ - type: object
+ required:
+ - schema
+ properties:
+ schema:
+ $ref: '#/components/schemas/Schema'
Review comment:
Are we defining this as always setting the current schema to the added schema? I think it's fine not too since we have the "set current schema" step below but I think we need to document that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org