You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/11/16 09:25:38 UTC
[GitHub] [iceberg] kbendick opened a new pull request #3561: [CORE] Specification for an HTTP REST catalog
kbendick opened a new pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561
Adds an OpenAPI definition for a proposal for an HTTP-based REST catalog.
Also converts it to markdown for easier viewing.
Needs a few examples filled in which I will do in the AM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751524099
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
Review comment:
I think this should be `/v1/namespaces`? Why the additional `/list`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751549178
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
Review comment:
It refers to v1 of the Rest API definition. I think we'd need to support v2 Iceberg spec (as well as v1 spec).
I'll find a place in the `info` section to make a note of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751573434
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
Review comment:
I believe you are right. There's not a need for that. I was working across a few copies and missed that `/v1/namespaces` isn't taken now, just `/v1/namespaces/{namespace}`.
I agree `/list` is likely not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r768117086
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
Review comment:
Ah ok. I see. I used it in editor. swagger.io and it didn't complain about it. But I can change it to updateProperties, since it's now really a mixture of both.
But I do like that it currently maps to existing API names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769860581
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
Review comment:
Oh actually I think you meant there wasn't a space after the comma.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765388331
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+
+ #############################
+ # Reusable Response Objects #
+ #############################
+ responses:
+
+ IcebergResponseObject:
Review comment:
Yeah that's the motivation.
Admittedly in some languages that's useful, in other's not so much. It also provides standardized form for the first top level of the message.
But you are right typically most APIs just send their response, as it's already a JSON object. I'd be fine to remove this, but it might also make the OpenAPI document definition a bit cleaner (though I'm open to any suggestions about using the OpenAPI document I might not be aware of if you have a lot of experience with using them).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r753883173
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
Review comment:
What's the use case for this? I'd say let's remove it if we don't need it in the initial implementation. I find it's usually good to add things when they are actually needed because you're closer to the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754718527
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
Review comment:
Is it possible to have a standard set of error codes, like the `400` and `401` here? Seems like those are possible for every request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754730663
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
Review comment:
I think that `POST /namespaces/{namespace}` to create is inconsistent with `POST /namespaces/{namespace}/tables` to create a table, where the table identifier is in the query. What about `POST /namespaces` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754729617
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
Review comment:
Error cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754731401
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
Review comment:
Should remove properties be done by DELETE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754634758
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
+ parameters:
+ - name: namespace
+ in: query
+ description: Namespace under which to list namespaces. Leave empty to list all namespaces in the catalog
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List all namespaces, or all namespaces underneat a given namespace
+ description: List namespaces underneath a given namespace
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ get:
+ tags:
+ - Catalog API
+ summary: Get the configured properties of a namespace
+ operationId: getNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ 'Application/JSON':
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
+ type: object
+ properties:
+ currentSnapshotTo:
+ $ref: '#/components/schemas/TableMetadata'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /v1
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+ NoSuchNamespaceError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request does not exist
+ NamespaceAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request already exists
+ NoSuchTableError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: The given table identifier does not exist
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+ IcebergResponseObject:
+ type: object
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ properties:
+ data:
+ $ref: '#/components/schemas/ResponseDataObject'
+ error:
+ $ref: '#/components/schemas/ResponseErrorObject'
+ example: "{ data: { }, error: { } }"
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body.
+ example: '{ data: { identifiers: [ "office.employees", "office.dogs", "office.cats" ] }'
+ ResponseErrorObject:
+ type: object
+ description: Error portion embedded in all JSON responses with further details of the error
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Machine-type of the errror, such as an exception class
+ code:
+ type: integer
+ description: Application specific error code, to disambiguage amongst subclasses of HTTP responses
+ example: 40101
+ metadata:
+ type: object
+ description: Additional metadata to accompany this error, such as server side stack traces or user instructions
+ nullable: true
+ required:
+ - message
+ - type
+ - code
+ examples:
+ catalogName:
+ value: prod
+ namespace:
+ value: prod.accounting
+ no_such_table_error:
+ value: |-
+ {
+ message: "The given table does not exist",
+ type: "NoSuchTableException",
+ code: 40101
+ }
+
+ no_such_iceberg_table_eror:
+ value: |-
+ {
+ message: "The given table does not exist",
Review comment:
Ok. Ignoring additional table types for the moment is probably easier for the first iteration. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754732448
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
Review comment:
Assuming this will be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757700129
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
Review comment:
There's a discussion elsewhere, but my concern with 404 is that in my experience, using 404 intentionally makes it hard to monitor a service (for example, many people monitor ELB response codes).
It's not possible from response code alone to disambiguate between expected 404 (e.g. namespace or other resouce doesn't exist, which isn't an operations concern generally speaking) and possibly misconfigured issues that people on-call would need to get paged for - like an unacceptable amount of 404s indicating they might need to check an nginx configuration or something to ensure that all the routes are being advertised appropriately.
There's still an open question of what response code to use instead, if any; I'm not entirely closed off to the idea to use 404.
But I have worked at places where our operations team eventually asked us to move off of intentional 404s so that they could monitor ELB response codes and get paged off of them without a fluctuating target for an acceptable number of 404s. And I have heard of similar efforts undertaken at Facebook etc, so I don't think it's unheard of to find using 404 undesirable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760541828
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
Review comment:
My understanding is that, at least per the spec for URIs, that individual components should be encoded after their separation character.
I wonder if any server frameworks would choke on a path parameter that essentially allows for [0, n] / (0, n] values separated by a slash? Having `tables` there makes it consistent with some of the other routes and provides a fully separator.
Namespace will already have to be encoded if it includes special characters, so I'm partial to the way I have it now, but if others feel strongly about your proposed idea, I'm certainly open to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760613253
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
Review comment:
Removed `tableLocation`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760617421
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
Unresolving this comment, as elsewhere it was suggested to use DELETE for removing properties. However, DELETE can't have a request body.
I can't find that original comment thread so reviving it here.
What verb / route do you think would work best for `removeProperties` from `SupportsNamespace`, given that `DELETE` cannot have a request body? cc @rdblue @jackye1995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-986426702
> > 412 feels weird. the fact that some people may have load balancers in front and some load balancers standard config aren't clear about what 404 means isn't a great reason for a non-standard overload of a return code.
>
> I agree with Ryan. I think the general principle of an OpenAPI specification is to express the theoretical behavior of the APIs, without consideration of infrastructure setup. There can be so many layers of infrastructure and redirection in between your service and the client, you can never predict what error could happen in between.
>
> ELB has a well-defined list of error codes, and counters for all of them, users can filter out what errors are from ELB and what are not if they really want to. https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html
>
> REST APIs are resources based, here the resources are `namespsace` and `table`. 404 of `/namespaces/ns` clearly indicates namespace `ns` does not exist, I don't think there is ambiguity related to that.
>
> 412 has a very specific meaning that "the server does not meet one of the preconditions that the requester put on the request header fields", we should not overload the meaning.
>
> I would suggest looking at some well-known APIs to see how they are defining error codes, such as https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html. Although S3 is not REST but RPC, but the error code aspect is the same.
That's a good point regarding infra complexities in between.
I always said I'd be fine with reverting to 404s, and I'm very much convinced now. I think there might be some use cases for additional code usage (e.g. a renameTable request where the namespace to rename the table to does not exist), but I'll revert everything to 404 for the foreseeable future. 👍
There are some places where using 404 multiple times is causing issues in generating the markdown, but that's more because a map key in a YAML needs to be unique. Not because we can't use 404 for those cases.
I've reverted everything to 404 (except one item I've kept as 409, while I figure out how to generate the markdown or shape the OpenAPI doc to only have one 404 key in the responses map).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r762731907
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
Review comment:
Oh yeah I can add the single quotes. In javascript, that would be accepted as a JSON object. So I meant for that to be a JSON object, but made the keys optionally strings as most JSON parsers allow that.
In the actual implementation, we'd use strings just to be sure. I can update that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r762817596
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
Review comment:
I have unfortunately not been able to find a standard way of documenting several common responses.
It seems like this should exist, so I'll keep my eyes out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763174894
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
Review comment:
Sounds like the right thing is a POST with a body that contains both properties to set and properties to remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-986426702
> > 412 feels weird. the fact that some people may have load balancers in front and some load balancers standard config aren't clear about what 404 means isn't a great reason for a non-standard overload of a return code.
>
> I agree with Ryan. I think the general principle of an OpenAPI specification is to express the theoretical behavior of the APIs, without consideration of infrastructure setup. There can be so many layers of infrastructure and redirection in between your service and the client, you can never predict what error could happen in between.
>
> ELB has a well-defined list of error codes, and counters for all of them, users can filter out what errors are from ELB and what are not if they really want to. https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html
>
> REST APIs are resources based, here the resources are `namespsace` and `table`. 404 of `/namespaces/ns` clearly indicates namespace `ns` does not exist, I don't think there is ambiguity related to that.
>
> 412 has a very specific meaning that "the server does not meet one of the preconditions that the requester put on the request header fields", we should not overload the meaning.
>
> I would suggest looking at some well-known APIs to see how they are defining error codes, such as https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html. Although S3 is not REST but RPC, but the error code aspect is the same.
That's a good point regarding infra complexities in between.
I always said I'd be fine with reverting to 404s, and I'm very much convinced now. I think there might be some use cases for additional code usage (e.g. a renameTable request where the namespace to rename the table to does not exist), but I'll revert everything to 404 for the foreseeable future. 👍
There are some places where using 404 multiple times is causing issues in generating the markdown, but that's more because a map key in a YAML needs to be unique. Not because we can't use 404 for those cases.
I've reverted everything to 404 in "Not Found" cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-970870654
Example of the Swagger editor view of the proposed API specification when copied into `editor.swagger.io`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] omarsmak commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
omarsmak commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766173652
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
My point is, from the description of this REST endpoint, this endpoint does two things, rename a table and move a table to new namespace. My problem with that, I find it a bit weird to include several actions in one REST request which could be misleading to the end user IMHO. Therefore, I was proposing if we could maybe break isolate these two functionalities and keep each by its own REST endpoint, also using what I proposed earlier and add a scope argument, we can limit this to move across namespaces, e.g:
```
PUT /namespaces/{namespace}/tables/{table}/actions?action=move&scope=namespace
PUT /namespaces/{namespace}/tables/{table}/actions?action=rename
```
This is just an idea to have for example a scope argument to only execute this action within that scope. So in the future, when we have move functionality for data, then the scope could be `?scope=data` for example. makes sense?
By the way, I think what I proposed earlier to have verbs in the REST URL may not go well with the REST guidelines, thus perhaps `actions` noun and the verb as argument would make more sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763240361
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
Review comment:
Updated all 5 digit codes to the three digit HTTP response codes. Will just rely on `type` for the explicit exception.
In the past, I've found it beneficial to have dedicated error codes one can look up. But looking back on it, that was more of a metrics optimization and to obscure information from the client side. That's not needed here, so type will be fine.
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 40401 }'
Review comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769879230
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
Review comment:
Oh. So the server side will / should check for these and return `422` (Unprocessable Entity).
In the Java implementation, there might be a way to detect these concurrent modifications early and throw, but I wouldn't want to over-specify in this area. In my opinion, it should be a route with well defined semantics and it should be up to the caller to craft a valid object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r756247388
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
Review comment:
That sounds reasonable to me. I think we should go with whatever is the normal or expected thing to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754633285
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
+ parameters:
+ - name: namespace
+ in: query
+ description: Namespace under which to list namespaces. Leave empty to list all namespaces in the catalog
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List all namespaces, or all namespaces underneat a given namespace
+ description: List namespaces underneath a given namespace
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ get:
+ tags:
+ - Catalog API
+ summary: Get the configured properties of a namespace
+ operationId: getNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ 'Application/JSON':
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
Review comment:
Not necessarily. It could be storing them in a database or even in memory. I would imagine that the rest server should guarantee that it keeps the metadata files up to date when they change, but it doesn't necessarily have to re-read the files if it can be sure that it's information is up to date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754716451
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
Review comment:
While I like the part about disabling certain operations, we don't need to include what we might be able to do in the future in the current docs. It's a great idea, but let's remove it from the description since it isn't currently possible.
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
Review comment:
Nit: I'd capitalize acronyms, like HTTP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-976924211
We were thinking more along the lines of the second option, having the REST API code generated from the Swagger definition. But I'm flexible as long as we have an OpenAPI source of truth and make sure we don't make accidental changes to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r753884218
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
Review comment:
I think `Unauthorized` is sufficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754735573
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
Review comment:
There are two of these so I'm not sure it was a mistake. Can you help me understand why this isn't `properties` and what it is for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754736509
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
+ type: object
+ properties:
+ currentSnapshotTo:
+ $ref: '#/components/schemas/TableMetadata'
+ IcebergConfiguration:
Review comment:
How about `CatalogConfiguration`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754466786
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
Review comment:
I agree with Jack. For consistency let's use `/v1/namespaces/{namespace}/tables`. We can consider adding a route for all tables later, but there is no corresponding method in the `Catlaog` API so I'm not sure that we need it. Let's concentrate on things in the exsting API for now to keep this small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754728793
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
Yeah, we should make this `GET /namespaces/{namespace}`. We can probably still use the `loadNamespaceMetadata` name. I like using the names that correspond to the interfaces.
(+1 for removing this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754723887
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
Review comment:
Looks like the example code is 5 digits in a few places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754724437
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
Review comment:
"the same catalog" assumes that there are multiple catalogs. I think we can remove that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754604277
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
Review comment:
I meant we can drop it from the summary. We should continue to have the `purgeRequested` flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760552335
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
Review comment:
I originally had a `POST /tables/renameTable` (similar to what you have here), but I changed it to this format at the request of somebody else.
Happy to change it back. I don't feel that the `namespace` for this path is super clear on the source one and the target one. I'd rather deal with two full identifiers specified in the body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760592555
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
Review comment:
Updated and moved to `DELETE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-984274681
> We were thinking more along the lines of the second option, having the REST API code generated from the Swagger definition. But I'm flexible as long as we have an OpenAPI source of truth and make sure we don't make accidental changes to it.
If we want to have the client generated from this document, we should probably use annotations in the code as mentioned by @nastra.
Even then, I don't feel the clients necessarily would respect all of the desired behavior we would want, particularly in the Java based catalog (though they could be "good enough" in a lot of places, particularly for curl commands and additional languages - either as a basis or just a client).
I have some of the annotations in another branch like @nastra demonstrated. More class definitions and things would need to be defined to add them in meaningfully, so I thought possibly that could be a next phase of this process if so desired.
The hardest part was figuring out a good way to generate even the OpenAPI doc from the annotations without using Spring etc. I saw some gradle tools but nothing stood out as canonical when I looked and I eventually just used Spring.
I can push some of them in another branch (this PR is already hard to open) and then if they provide value we can move forward with them. If we want to rely on this document (and not just use it for discussion), we should definitely consider the annotations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757705583
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
+
+<h1 id="apache-iceberg-rest-catalog-api-configuration-api">Configuration API</h1>
+
+## getConfig
+
+<a id="opIdgetConfig"></a>
+
+> Code samples
+
+```shell
+# You can also use wget
+curl -X GET http://127.0.0.1:1080/v1/config \
+ -H 'Accept: application/json' \
+ -H 'Authorization: Bearer {access-token}'
+
+```
+
+```java
+URL obj = new URL("http://127.0.0.1:1080/v1/config");
+HttpURLConnection con = (HttpURLConnection) obj.openConnection();
+con.setRequestMethod("GET");
+int responseCode = con.getResponseCode();
+BufferedReader in = new BufferedReader(
+ new InputStreamReader(con.getInputStream()));
+String inputLine;
+StringBuffer response = new StringBuffer();
+while ((inputLine = in.readLine()) != null) {
+ response.append(inputLine);
+}
+in.close();
+System.out.println(response.toString());
+
+```
+
+`GET /v1/config`
+
+*List all catalog configuration settings*
+
+All REST catalogs will be initialized by calling this route. This route will return at least the minimum necessary metadata to initialize the catalog. Optionally, it can also include server-side specific overrides. For example, it might also include information used to initialize this catalog such as the details of the Http connection pooling, etc. This route might also advertise information about operations that are not implemented so that the catalog can eagerly throw or go about another way of performing the desired action.
Review comment:
Certainly I don't mind clarifying, thank you for asking. I will also update the description. :D
So connection pooling is admittedly a bad example, as you're right that it is really application specific and best placed with the application. Though admittedly, even that some people might want to be able to control.
> Or is this meant more for the Spark (or equivalent) catalog settings?
Ideally it's meant for that first and foremost.
Having worked in managing a large number of people's jobs, where often times they didn't care about the underlying architecture or want to have to deal with it), there are some Spark configs, s3 configs, and possibly even HTTP client configs it would be helpful to be able to inject into user's jobs as in some settings it's hard to get all users to put the right things in the right places.
So things like Spark configs can definitely fit into that scope and originally that was my thinking for this endpoint.
> Or is the idea that the http client would get re-configured once this call is made.
Outside of Spark configs, I'm not opposed to the idea of "re-configuring", though I wouldn't necessarily re-configure but have a thin single use HTTP client that is started on the driver or equivalent for one initial setup, that is then closed and the real HTTP client takes over.
Then, the REST catalog can be fully `initialize`d itself based off a combination of user provided information and information from the server side. One place where that's common is users use temporary AWS IAM role credentials.
I'm open to both, I guess. I was admittedly more thinking like engine-catalog specific configurations at first.
For example, Flink requires catalog's to have known configs when building the catalog and this might be one way to get them somewhat more dynamically than just adding them all to the source code one by one which might not make sense in all scenarios. Or even think of the number of s3a configs there are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r753884376
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
Review comment:
I see the note above for 404. That sounds reasonable to me, but we should consider other options for this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754724269
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
Review comment:
409 seems reasonable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754717129
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
Review comment:
By _first_ calling this route. Also, what about updating to "All REST catalog clients will first call this route to get configuration provided by the server"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754579043
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
Review comment:
Hmmm.... I'll take a look. I think we can provide example responses and then a description to that effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754456983
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
Review comment:
Is there a place to document that this is true if the table was dropped and false if the table did not exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754733562
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
Review comment:
Can you also request a specific `location`? That seems reasonable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766101374
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
Review comment:
I had not heard that before. Do you have any source indicating that?
I updated this to use the slash because this one route is handling both `setPrpoperties` call and `removeProperties` call: https://github.com/apache/iceberg/blob/59a76da2ad78b3fabed024baa359e931d239ddc9/api/src/main/java/org/apache/iceberg/catalog/SupportsNamespaces.java#L107-L129
editor.swagger.io did not complain (and it seems to complain about a number of other things), but if what you're is true, then probably the operationId could be `setOrRemoveProperties` or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r768099937
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
Review comment:
You mean the period, right? Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765384883
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
Review comment:
It really is just a map of properties though, at least as far as the existing usage of namespace's properties considers it.
We could possibly add some specific properties onto it, but we'd have to allow for general string key-value map entries still.
What entries would you have in mind? I don't believe there are any standard properties for a namespace. For table's and for catalog's yes, but not for namespaces as far as I'm aware.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769887769
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
Review comment:
Updated to 400.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
snazy commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765539500
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
Review comment:
Is this really needed in the spec?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
Review comment:
`name` missing?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
Review comment:
Not sure whether it's useful to have a server/url definition in the spec for Iceberg. This particular one makes it mandatory to use 3 variables, which makes it complicated to "just use" the correct URL provided by the (web/REST) server implementation.
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configured properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
Review comment:
Wouldn't it be easier to just return the full set of properties instead of splitting it up into updated, removed and a maybe not present `notPresent`?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configured properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
Review comment:
Typo: `Servers`
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+
+ #############################
+ # Reusable Response Objects #
+ #############################
+ responses:
+
+ IcebergResponseObject:
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ content:
+ application/json:
+ schema:
+ anyOf:
+ - $ref: '#/components/schemas/ResponseDataObject'
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ example: { "data": { "namespaces": [ [ "ns1", "foo_db" ], [ "ns1", "bar_db" ] ] }, "error": { } }
Review comment:
Feels like a `oneOf` would effectively make `IcebergResponseObject` superfluous and operations can specify the "right" type for each response without an intermediate `ResponseDataObject` or `ResponseErrorObject`?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configured properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
Review comment:
What's the intention of this rootPath/prefix property?
Mean, you need a proper URL with the right prefix to get the IcebergConfiguration object, so would this rootPath/prefix point to itself?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+
+ #############################
+ # Reusable Response Objects #
+ #############################
+ responses:
+
+ IcebergResponseObject:
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ content:
+ application/json:
+ schema:
+ anyOf:
+ - $ref: '#/components/schemas/ResponseDataObject'
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ example: { "data": { "namespaces": [ [ "ns1", "foo_db" ], [ "ns1", "bar_db" ] ] }, "error": { } }
Review comment:
I think if ResponseDataObject/ResponseErrorObject just contain a generic JSON doc/object, clients have to manually construct the right object type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765382480
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
Review comment:
Yes.
Some of these so far, to make reviewing easier earlier on, I left the response a simple Ok to make it slightly less wordy as people focused on other parts. I will update this one. Good catch.
All routes that can return a body, will likely return a body to allow for passing metadata down and for evolving in the future. Of course things like HEAD won't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766208929
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
Review comment:
It wouldn't be very useful to use the Swagger UI to make requests if we couldn't define the host etc.
I don't know how one would use the swagger UI to make calls without having a server variable, given that people's host will surely be different.
It was @rymurr who asked me to make it configurable in a comment that is.... somewhere deep within all of the comments of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766098992
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
Review comment:
If it were the case that `error` and `data` were both allowed (for example to represent partial failure / partial success), then maybe this would go to 200.
But yeah, most likely we don't expect this to go below 400.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757705870
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
+
+<h1 id="apache-iceberg-rest-catalog-api-configuration-api">Configuration API</h1>
+
+## getConfig
+
+<a id="opIdgetConfig"></a>
+
+> Code samples
+
+```shell
+# You can also use wget
+curl -X GET http://127.0.0.1:1080/v1/config \
+ -H 'Accept: application/json' \
+ -H 'Authorization: Bearer {access-token}'
+
+```
+
+```java
+URL obj = new URL("http://127.0.0.1:1080/v1/config");
+HttpURLConnection con = (HttpURLConnection) obj.openConnection();
+con.setRequestMethod("GET");
+int responseCode = con.getResponseCode();
+BufferedReader in = new BufferedReader(
+ new InputStreamReader(con.getInputStream()));
+String inputLine;
+StringBuffer response = new StringBuffer();
+while ((inputLine = in.readLine()) != null) {
+ response.append(inputLine);
+}
+in.close();
+System.out.println(response.toString());
+
+```
+
+`GET /v1/config`
+
+*List all catalog configuration settings*
+
+All REST catalogs will be initialized by calling this route. This route will return at least the minimum necessary metadata to initialize the catalog. Optionally, it can also include server-side specific overrides. For example, it might also include information used to initialize this catalog such as the details of the Http connection pooling, etc. This route might also advertise information about operations that are not implemented so that the catalog can eagerly throw or go about another way of performing the desired action.
Review comment:
TLDR - I'll update the example description for sure as the connection pooling is pretty application specific 👍
Though we might want to allow for a one time call to get that sort of information from more of a data operations team as analysts and other end users might not want to have to deal with configuring everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757589664
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
Review comment:
specifying base url and port as part of the spec/docs seems strong. Shouldn't this be config?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
Review comment:
404 if namespace doesn't exist?
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
+
+<h1 id="apache-iceberg-rest-catalog-api-configuration-api">Configuration API</h1>
+
+## getConfig
+
+<a id="opIdgetConfig"></a>
+
+> Code samples
+
+```shell
+# You can also use wget
+curl -X GET http://127.0.0.1:1080/v1/config \
+ -H 'Accept: application/json' \
+ -H 'Authorization: Bearer {access-token}'
+
+```
+
+```java
+URL obj = new URL("http://127.0.0.1:1080/v1/config");
+HttpURLConnection con = (HttpURLConnection) obj.openConnection();
+con.setRequestMethod("GET");
+int responseCode = con.getResponseCode();
+BufferedReader in = new BufferedReader(
+ new InputStreamReader(con.getInputStream()));
+String inputLine;
+StringBuffer response = new StringBuffer();
+while ((inputLine = in.readLine()) != null) {
+ response.append(inputLine);
+}
+in.close();
+System.out.println(response.toString());
+
+```
+
+`GET /v1/config`
+
+*List all catalog configuration settings*
+
+All REST catalogs will be initialized by calling this route. This route will return at least the minimum necessary metadata to initialize the catalog. Optionally, it can also include server-side specific overrides. For example, it might also include information used to initialize this catalog such as the details of the Http connection pooling, etc. This route might also advertise information about operations that are not implemented so that the catalog can eagerly throw or go about another way of performing the desired action.
Review comment:
Could you clarify this a bit more for me? Maybe I am daft but it seems a bit circular to me. We need things like the url, auth, http client settings etc to call this endpoint but it returns the config we need to connect? Or is the idea that the http client would get re-configured once this call is made. Additionally, settings like the connection pooling sound like a concern of the client rather than the server (eg a lambda isn't going to use connection pooling whilst a long running service may use it a lot).
Or is this meant more for the Spark (or equivalent) catalog settings?
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
Review comment:
Same goes for authentication. I would expect auth would have to be pluggable in the spirit of `FileIO` rather than making the spec only support http bearer auth.
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
Review comment:
just to be clear this is a new exception class that would be http server/clietn specific. Not the existing exception. Thats correct?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
Review comment:
Why not just `v1/namespaces/{namespace}/{namespace}/{table}`? specifically why the extra `tables`?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
Review comment:
why not use path separators as namespace separators.
Also, what happens if the namespace has a `.` in it or other url characters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757614046
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
Review comment:
Why not just v1/namespaces/{namespace}/{namespace}/{table}? specifically why the extra tables?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
Review comment:
I wonder if the delete should be here as well? Similar reason as the method above. Dropping a table (to me) can be construed as a transaction and that is a job for the table ops to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760678580
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
I moved `removeProperties` back to `PUT` for now, which can have a request body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760595501
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
Review comment:
That's correct. Sorry I wrote a response to this but somehow it didn't go through.
This is the error schema (so the JSON wrapper for the response), but it is `NoSuchTableException`.
It would in theory look like the following, where `code` is application defined and can be looked up (open to removing that part but it has been useful at other organizations I've been in to have more specific error codes in the response):
```
{ "error": { "message": "The given table does not exist", "type": "NoSuchTableException", "code": 41201 } }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760684704
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
Review comment:
So these markdown files are autogenerated by a separate tool.
For now, I just stuck bearer token in there as a means of indicating some kind of HTTP Authentication header at the least. I'll look into how to add a configurable section to the OpenAPI document itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763137235
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
Review comment:
I agree that having PUT and POST is a bit confusing. The reason I switched to PUT here is that a `DELETE` request isn't supposed to have a request body.
I could likely model all of this into one POST request though, where the instructions are in the body or not.
Let me do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
nastra commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763183617
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
Review comment:
agreed, POST definitely seems to be the right option here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751499966
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
Review comment:
I think we don't need a explicit `renameTable` path, rename table can be expressed as a part of PUT `/tables/{table}` where the update payload is a different table name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-970870654
Example of the Swagger editor view of the proposed API specification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751493263
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
Review comment:
Good call. Consistency is good. I'm definitely open to updating the routes as needed for consistency and to avoid any potential issues with generating the requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763123797
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
Review comment:
Yeah. I'll update those. The extended code is meant to signify further information than just the response code, but we already have `type` for that.
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
Review comment:
Moved to 404.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763125591
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 40401 }'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
Review comment:
You're absolutely right. I don't know why the swagger editor I was working with complained about responses on other things but not HEAD. Updated to just 200 and 404. Happy to do 204 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754734959
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
Review comment:
Why not name this `properties`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754734678
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
Review comment:
`NamespaceProperties`? It may make sense to define `Namespace` as a list of strings. Or maybe `NamespaceIdentifier`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754732813
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
Review comment:
Looks like this isn't used. Probably stale from the old version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
nastra commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-976231559
> > Have you thought about/considered something like https://github.com/eclipse/microprofile-open-api (just an example, but there are other libs available as well), which allows you to add OpenAPI annotations directly to your Java code, so that the OpenAPI spec+docs and the code are all at one place?
>
> Right now, the main thing is to have something to discuss, and being able to paste this file into the swagger UI makes it really easy to discuss the API.
Agreed that having the pure Swagger API definition for discussion is probably the best. I was rather thinking long term and how we want to approach that.
>
> Longer term, I'm not sure what the best path will be to maintain the API definition, but I'm really reluctant to publish whatever happens to be in the Java implementation through annotations. While that's an easy way to keep the code and the definition in sync, it is probably not a great way to manage a specification that is portable. It's too easy to refactor Java and make unintended changes to the OpenAPI spec, right? Maybe I'm wrong here, but it seems easy to mess up the API that way.
It can be indeed easy to change the OpenAPI spec by changing/refactoring Java stuff. An approach to mitigating this would be to generate the Swagger API definition to a folder that is version-controlled and always commit that API definition as well. That way we could spot those changes.
Alternatively, we could have the REST API code be generated from the Swagger definition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754733801
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
Review comment:
I don't think this should be in the request, since it could differ from the individual objects (e.g., schema).
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
Review comment:
TBD?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
nastra commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-975539916
Have you thought about/considered something like https://github.com/eclipse/microprofile-open-api (just an example, but there are other libs available as well), which allows you to add OpenAPI annotations directly to your Java code, so that the OpenAPI spec+docs and the code are all at one place?
In combination with JAX-RS annotations, everything that defines a REST endpoint would be combined at a single place and can be easier consumed/reasoned about by developers, rather than having the Swagger definitions and the endpoint implementations across multiple files.
The `swagger.json|yaml` is then generated from the annotations, which we could then serve by a separate REST endpoint (it would basically show what you see when you upload the `swagger.yaml` to editor.swagger.io).
Below is a quick & short example that shows how this would look like in code:
```
@Path("config")
public interface ConfigurationApi {
@GET
@Produces(MediaType.APPLICATION_JSON)
@Operation(summary = "List all catalog configuration settings")
@APIResponses({
@APIResponse(
responseCode = "200",
description =
"All REST catalogs will be initialized by calling this route. This route\n"
+ "will return at least the minimum necessary metadata to initialize the\n"
+ "catalog. Optionally, it can also include server-side specific overrides.\n"
+ "For example, it might also include information used to initialize this catalog\n"
+ "such as the details of the Http connection pooling, etc. This route might\n"
+ "also advertise information about operations that are not implemented\n"
+ "so that the catalog can eagerly throw or go about another way of performing\n"
+ "the desired action.",
content =
@Content(
mediaType = MediaType.APPLICATION_JSON,
examples = {@ExampleObject(ref = "configurationExample")},
schema = @Schema(implementation = IcebergConfiguration.class))),
@APIResponse(responseCode = "400", description = "Unknown Error"),
@APIResponse(responseCode = "401", description = "Invalid credentials provided"),
})
IcebergConfiguration getConfig();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-975833631
> Have you thought about/considered something like https://github.com/eclipse/microprofile-open-api (just an example, but there are other libs available as well), which allows you to add OpenAPI annotations directly to your Java code, so that the OpenAPI spec+docs and the code are all at one place?
Right now, the main thing is to have something to discuss, and being able to paste this file into the swagger UI makes it really easy to discuss the API.
Longer term, I'm not sure what the best path will be to maintain the API definition, but I'm really reluctant to publish whatever happens to be in the Java implementation through annotations. While that's an easy way to keep the code and the definition in sync, it is probably not a great way to manage a specification that is portable. It's too easy to refactor Java and make unintended changes to the OpenAPI spec, right? Maybe I'm wrong here, but it seems easy to mess up the API that way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754458896
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
Review comment:
You can drop "within a given namespace" since that's implied by the required parameter.
Can you also remove the notes from the description? I don't think your intent was to leave "Can change to returning..." in the docs, right? We should have only one "right" way to implement this protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754454754
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
Review comment:
Can you remove this operation? This is part of `TableOperations`, not `Catalog` and I don't think we want to dive into table updates in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754633285
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
+ parameters:
+ - name: namespace
+ in: query
+ description: Namespace under which to list namespaces. Leave empty to list all namespaces in the catalog
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List all namespaces, or all namespaces underneat a given namespace
+ description: List namespaces underneath a given namespace
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ get:
+ tags:
+ - Catalog API
+ summary: Get the configured properties of a namespace
+ operationId: getNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ 'Application/JSON':
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
Review comment:
Not necessarily. It could be storing them in a database or even in memory in a local cache.
I would imagine that the rest server should guarantee that it keeps the metadata files up to date when they change, but it doesn't necessarily have to re-read the files if it can be sure that the metadata.json file information it has stored is up to date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760547087
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
Review comment:
I've worked at some companies where our operations team has asked us to change off of explicitly returned 404.
I agree that services should have metrics etc, but many load balancers themselves cant be configured with this additional information. It often has to go within user's applications. That said, I've definitely been in outage situations where more custom metrics / app metrics are down, and one of the few signals left is raw ELB response codes (which can usually be retrieved outside of normal in-app metrics systems people use).
There's also precedence for departing from standard REST conventions for internal monitring. For example, for all "valid" requests, Facebook strictly returns a 2xx for monitoring purposes.
I'm open to using 404, but I don't personally subscribe to the belief that we should ignore making internal monitoring easier for the sake of purity to the REST standard. Additionally, if we had 412 and 409, we'd be able to easily see what caused the "404" type response. Plain 404 everywhere wouldn't indicate what was missing without further metrics.
I'm not closed off to using 404, but that's my reasoning. I think it's healthier to try to work with whatever internal teams might be responsible for monitoring and to allow as many avenues as possible to determine what went wrong, especially if we presume that inevitably certain metrics might temporarily not be available when we need them most. So there's precedence both ways.
I'm going to leave it for now but am happy to continue the discussion 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760603898
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
I'm going to resolve this discussion now that we've moved loadNamespaceMetadata outside of the `properties` route and have standardized the `setProperties` and `removeProperties` endpoints a bit.
Please feel free to reopen if you have concerns!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-984191129
> Have you thought about/considered something like https://github.com/eclipse/microprofile-open-api (just an example, but there are other libs available as well), which allows you to add OpenAPI annotations directly to your Java code, so that the OpenAPI spec+docs and the code are all at one place? In combination with JAX-RS annotations, everything that defines a REST endpoint would be combined at a single place and can be easier consumed/reasoned about by developers, rather than having the Swagger definitions and the endpoint implementations across multiple files. The `swagger.json|yaml` is then generated from the annotations, which we could then serve by a separate REST endpoint (it would basically show what you see when you upload the `swagger.yaml` to editor.swagger.io).
I implemented some of these in a separate project (where I added Spring, as I couldn't figure out how to generate the OpenAPI doc from the eclipse microapi annotations - though they're essentially the same).
It might make more sense to add it in a second pass when there's a bit more implemented. Presently it didn't feel like it provided a ton of value (and I'd need to figure out how to get it into the Iceberg build setup, as I used a separate project eventually so I could use Spring), though I can see how it could in the longer term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760550740
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
Review comment:
Removed anyway since I got rid of pagination! But good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760572030
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
+
+<h1 id="apache-iceberg-rest-catalog-api-configuration-api">Configuration API</h1>
+
+## getConfig
+
+<a id="opIdgetConfig"></a>
+
+> Code samples
+
+```shell
+# You can also use wget
+curl -X GET http://127.0.0.1:1080/v1/config \
+ -H 'Accept: application/json' \
+ -H 'Authorization: Bearer {access-token}'
+
+```
+
+```java
+URL obj = new URL("http://127.0.0.1:1080/v1/config");
+HttpURLConnection con = (HttpURLConnection) obj.openConnection();
+con.setRequestMethod("GET");
+int responseCode = con.getResponseCode();
+BufferedReader in = new BufferedReader(
+ new InputStreamReader(con.getInputStream()));
+String inputLine;
+StringBuffer response = new StringBuffer();
+while ((inputLine = in.readLine()) != null) {
+ response.append(inputLine);
+}
+in.close();
+System.out.println(response.toString());
+
+```
+
+`GET /v1/config`
+
+*List all catalog configuration settings*
+
+All REST catalogs will be initialized by calling this route. This route will return at least the minimum necessary metadata to initialize the catalog. Optionally, it can also include server-side specific overrides. For example, it might also include information used to initialize this catalog such as the details of the Http connection pooling, etc. This route might also advertise information about operations that are not implemented so that the catalog can eagerly throw or go about another way of performing the desired action.
Review comment:
I think we need to be really clear about what can be returned here and what the requirements are. This is similar to my question about whether these are defaults or overrides. We definitely want to set clear expectations.
I think we probably want:
* Settings from the server to be used as defaults, unless we want to support both defaults and overrides. Explicit local config should take priority, or else users would be unable to make changes.
* This should be called first in initialization and all real initialization should happen with the updated properties.
* The connection used to fetch server config should be thrown away so that all of the returned defaults are used for subsequent connections
Now that I'm thinking about it more, I think we will want two sets, defaults and overrides. That way the server can do things like send a session token to replace the original credentials or force certain behaviors when something isn't supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757705870
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
+
+<h1 id="apache-iceberg-rest-catalog-api-configuration-api">Configuration API</h1>
+
+## getConfig
+
+<a id="opIdgetConfig"></a>
+
+> Code samples
+
+```shell
+# You can also use wget
+curl -X GET http://127.0.0.1:1080/v1/config \
+ -H 'Accept: application/json' \
+ -H 'Authorization: Bearer {access-token}'
+
+```
+
+```java
+URL obj = new URL("http://127.0.0.1:1080/v1/config");
+HttpURLConnection con = (HttpURLConnection) obj.openConnection();
+con.setRequestMethod("GET");
+int responseCode = con.getResponseCode();
+BufferedReader in = new BufferedReader(
+ new InputStreamReader(con.getInputStream()));
+String inputLine;
+StringBuffer response = new StringBuffer();
+while ((inputLine = in.readLine()) != null) {
+ response.append(inputLine);
+}
+in.close();
+System.out.println(response.toString());
+
+```
+
+`GET /v1/config`
+
+*List all catalog configuration settings*
+
+All REST catalogs will be initialized by calling this route. This route will return at least the minimum necessary metadata to initialize the catalog. Optionally, it can also include server-side specific overrides. For example, it might also include information used to initialize this catalog such as the details of the Http connection pooling, etc. This route might also advertise information about operations that are not implemented so that the catalog can eagerly throw or go about another way of performing the desired action.
Review comment:
**TLDR** - I'll update the example description for sure as the connection pooling is pretty application specific 👍
Though we might want to allow for a one time call to get that sort of information from more of a data operations team as analysts and other end users might not want to have to deal with configuring everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757705583
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
+
+<h1 id="apache-iceberg-rest-catalog-api-configuration-api">Configuration API</h1>
+
+## getConfig
+
+<a id="opIdgetConfig"></a>
+
+> Code samples
+
+```shell
+# You can also use wget
+curl -X GET http://127.0.0.1:1080/v1/config \
+ -H 'Accept: application/json' \
+ -H 'Authorization: Bearer {access-token}'
+
+```
+
+```java
+URL obj = new URL("http://127.0.0.1:1080/v1/config");
+HttpURLConnection con = (HttpURLConnection) obj.openConnection();
+con.setRequestMethod("GET");
+int responseCode = con.getResponseCode();
+BufferedReader in = new BufferedReader(
+ new InputStreamReader(con.getInputStream()));
+String inputLine;
+StringBuffer response = new StringBuffer();
+while ((inputLine = in.readLine()) != null) {
+ response.append(inputLine);
+}
+in.close();
+System.out.println(response.toString());
+
+```
+
+`GET /v1/config`
+
+*List all catalog configuration settings*
+
+All REST catalogs will be initialized by calling this route. This route will return at least the minimum necessary metadata to initialize the catalog. Optionally, it can also include server-side specific overrides. For example, it might also include information used to initialize this catalog such as the details of the Http connection pooling, etc. This route might also advertise information about operations that are not implemented so that the catalog can eagerly throw or go about another way of performing the desired action.
Review comment:
Certainly I don't mind clarifying, thank you for asking. I will also update the description. :D
So connection pooling is admittedly a bad example, as you're right that it is really application specific and best placed with the application. Though admittedly, even that some people might want to be able to control.
> Or is this meant more for the Spark (or equivalent) catalog settings?
Ideally it's meant for that first and foremost.
Having worked in managing a large number of people's jobs, where often times they didn't care about the underlying architecture or want to have to deal with it), there are some configs it's nice to be able to inject oneself vs having to get all of your users to put them there. So things like Spark configs can definitely fit into that scope.
> Or is the idea that the http client would get re-configured once this call is made.
Outside of Spark configs, I'm not opposed to the idea of "re-configuring", though I wouldn't necessarily re-configure but have a thin single use HTTP client that is started on the driver or equivalent for one initial setup, that is then closed and the real HTTP client takes over.
Then, the REST catalog can be fully `initialize`d itself off of that.
So mostly, yes Spark configs or even Iceberg specific configs that are somewhat terse and a pain for users but a good idea in a particular environment as a default. So probably leaving it as a thin client that is closed as an init type situation might make sense to allow the catalog and the HTTP client to be more fully configured. Off the top of my head, the user might have an authentication token and a base address, but this endpoint could then provide them the correct API endpoint / host for their specific environment - e.g. maybe a dev vs prod type set up (as one example). In which case this endpoint would need to be called so that the full http client can be configured.
I'm open to both, but I was admittedly more thinking like engine-catalog specific configurations, like Flink requires catalogs to have known configs and this might be one way to get them somewhat more dynamically than just adding them all to the source code one by one which might not make sense in all scenarios (think of the number of s3a configs there are even).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-984945739
> 412 feels weird. the fact that some people may have load balancers in front and some load balancers standard config aren't clear about what 404 means isn't a great reason for a non-standard overload of a return code.
I agree with Ryan. I think the general principle of an OpenAPI specification is to express the theoretical behavior of the APIs, without consideration of infrastructure setup. There can be so many layers of infrastructure and redirection in between your service and the client, you can never predict what error could happen in between.
ELB has a well-defined list of error codes, and counters for all of them, users can filter out what errors are from ELB and what are not if they really want to. https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html
REST APIs are resources based, here the resources are `namespsace` and `table`. 404 of `/namespaces/ns` clearly indicates namespace `ns` does not exist, I don't think there is ambiguity related to that.
412 has a very specific meaning that "the server does not meet one of the preconditions that the requester put on the request header fields", we should not overload the meaning.
I would suggest looking at some well-known APIs to see how they are defining error codes, such as https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html. Although S3 is not REST but RPC, but the error code aspect is the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766092172
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
Oh actually, do you mean `move` as in move the underlying data files for a table to another location?
That's not currently supported by the catalog interface as far as I am aware at the moment.
For HadoopCatalog, [`renameTable` isn't supported currently ](https://github.com/apache/iceberg/blob/f68d8d426661efc0d7e5686fe833b573b74eadab/core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCatalog.java#L237-L246 ) and it throws `UnsupportedOperationException` as the table would need to be moved.
For HiveCatalog, `renameTable` results [in the same location for the new identifier](https://github.com/apache/iceberg/blob/827b6c86108eec7b6de25f61022c7f8b5dda481c/hive-metastore/src/test/java/org/apache/iceberg/hive/HiveTableTest.java#L121) as the original table being renamed (so not a move by this definition of move).
The could be a good addition, but I think it's tangential to this very initial PR (that's more intended to cover the minimum subset of things for existing behavior, specifically for `namespaces`), while still remaining possible to cover new behavior in the future (or even in future iterations of this initial specification).
I do think `move` could possibly be a good addition, but that should likely be handled in another PR.
Is that what you meant by `move`? To move the data? @omarsmak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dimas-b commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
dimas-b commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766032201
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
IMHO, `type` is preferable to numeric `code`, but it might be harder to define strictly and there's risk that `type` names deviate from actual exception class names. Plus fixing class names in the API spec might affect non-java implementations in a negative way and complicate java code refactoring too.
In Nessie, as an example, we have an enum of non-numeric error codes. The API returns enum names, but they can be mapped to java exceptions upon parsing. Python client has its own mapping too. cf. [ErrorCode.java](https://github.com/projectnessie/nessie/blob/main/model/src/main/java/org/projectnessie/error/ErrorCode.java)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dimas-b commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
dimas-b commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766164043
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
Review comment:
Well, if we approach it from the other end (to try and understand this better), what is the intended use case for `defaults`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763238950
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
Review comment:
I updated this to a single POST request. It has three parameters:
- Path parameters: Namespace - URI encoded namespace
- Request body parameters:
- toUpdate: Object containing key-value pairs to upsert
- toRemove: Array (set) of keys to remove from the properties
It would be an invalid request to specify a key from toRemove in toUpdate as well (need to add that in).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751502851
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
Review comment:
I think this should be `/v1/namespaces/{namespace}/tables`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754724076
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
Review comment:
Let's use 412 and only have a comment in one place (maybe reference it?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754717606
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
Review comment:
What do you mean by "at least the minimum necessary metadata to initialize"? I can't see this providing the base URL of the service, for example. And, I don't think that this is a necessary requirement for services. They should be able to rely on some client config, like a token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754455868
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
Review comment:
Might want to change this to `purgeRequested`. I don't think there is a requirement to purge, but we do want to pass it if the engine did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754733149
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
Review comment:
Should the example be a valid JSON object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754728300
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
Review comment:
This doesn't seem right. The description is "The server cannot meet the requirements of the Expect request-header field." What about 412 like the table routes for `NoSuch*Exception`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754628089
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
Review comment:
The purge option also changes the semantics of the API, as drop could be synchronous and purge might make the API asynchronous and return 202 instead of 200.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751568503
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
I added this one to be able to mirror this API of `loadNamespaceMetadata`: https://github.com/apache/iceberg/blob/f96f06ea07841c137a98a25e4edf62c508475801/api/src/main/java/org/apache/iceberg/catalog/SupportsNamespaces.java#L89-L96
Right now, between the two interfaces `Catalog` and `SupportsNamespaces`, I don't think there's a method that just returns a namespace. Though a namespace at its minimum is really just a combination of its properties and its name.
In terms of the current API for returning the details of a namespace, the following exists:
1) listing the namespaces underneath a namespace (or all namespaces)
2) listing the tables underneath a namespace
3) getting the properties of a namespace.
The other reason I added the `properties` route is to be able to support `setProperties` and `removeProperties`: https://github.com/apache/iceberg/blob/f96f06ea07841c137a98a25e4edf62c508475801/api/src/main/java/org/apache/iceberg/catalog/SupportsNamespaces.java#L107-L129
I'm still very open to making changes that seem appropriate, especially as you have more experience writing catalogs than I do. What do you think knowing that we'd need to support updating / removing a subset of properties (e.g. `setProperties` and `removeProperties`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751551087
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
Review comment:
I originally had it that way, but the one use case that I wanted to be able to support was listing all tables, regardless of namespace.
That's why for this path, I have `namespace` as a query parameter. I don't love the inconsistency though so would appreciate any thoughts on how to support both while being as consistent as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769856626
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
These have been moved into one POST call which essentially is `updateProperties`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757705583
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
+
+<h1 id="apache-iceberg-rest-catalog-api-configuration-api">Configuration API</h1>
+
+## getConfig
+
+<a id="opIdgetConfig"></a>
+
+> Code samples
+
+```shell
+# You can also use wget
+curl -X GET http://127.0.0.1:1080/v1/config \
+ -H 'Accept: application/json' \
+ -H 'Authorization: Bearer {access-token}'
+
+```
+
+```java
+URL obj = new URL("http://127.0.0.1:1080/v1/config");
+HttpURLConnection con = (HttpURLConnection) obj.openConnection();
+con.setRequestMethod("GET");
+int responseCode = con.getResponseCode();
+BufferedReader in = new BufferedReader(
+ new InputStreamReader(con.getInputStream()));
+String inputLine;
+StringBuffer response = new StringBuffer();
+while ((inputLine = in.readLine()) != null) {
+ response.append(inputLine);
+}
+in.close();
+System.out.println(response.toString());
+
+```
+
+`GET /v1/config`
+
+*List all catalog configuration settings*
+
+All REST catalogs will be initialized by calling this route. This route will return at least the minimum necessary metadata to initialize the catalog. Optionally, it can also include server-side specific overrides. For example, it might also include information used to initialize this catalog such as the details of the Http connection pooling, etc. This route might also advertise information about operations that are not implemented so that the catalog can eagerly throw or go about another way of performing the desired action.
Review comment:
Certainly I don't mind clarifying, thank you for asking. I will also update the description. :D
So connection pooling is admittedly a bad example, as you're right that it is really application specific and best placed with the application. Though admittedly, even that some people might want to be able to control.
> Or is this meant more for the Spark (or equivalent) catalog settings?
Ideally it's meant for that first and foremost.
Having worked in managing a large number of people's jobs, where often times they didn't care about the underlying architecture or want to have to deal with it), there are some configs it's nice to be able to inject oneself vs having to get all of your users to put them there. So things like Spark configs can definitely fit into that scope.
> Or is the idea that the http client would get re-configured once this call is made.
Outside of Spark configs, I'm not opposed to the idea of "re-configuring", though I wouldn't necessarily re-configure but have a thin single use HTTP client that is started on the driver or equivalent for one initial setup, that is then closed and the real HTTP client takes over.
Then, the REST catalog can be fully `initialize`d itself off of that.
So mostly, yes Spark configs or even Iceberg specific configs that are somewhat terse and a pain for users but a good idea in a particular environment as a default. So probably leaving it as a thin client that is closed as an init type situation might make sense to allow the catalog and the HTTP client to be more fully configured. Off the top of my head, the user might have an authentication token and a base address, but this endpoint could then provide them the correct API endpoint / host for their specific environment - e.g. maybe a dev vs prod type set up (as one example). In which case this endpoint would need to be called so that the full http client can be configured.
I'm open to both, I guess. I was admittedly more thinking like engine-catalog specific configurations, like Flink requires catalogs to have known configs and this might be one way to get them somewhat more dynamically than just adding them all to the source code one by one which might not make sense in all scenarios (think of the number of s3a configs there are even).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760632701
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
Review comment:
I wouldn't mind removing it. I'll hold off until other's chime in, who might have more experience in using OpenAPI to version their API spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760678956
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
Review comment:
Removed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760613477
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
Review comment:
Yeah I will update it and then add a concrete example as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] omarsmak commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
omarsmak commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r764687471
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
I am concerned here to mix the two functionalities into one method. Why don't we just separate these two functionalities and move under the table endpoint? E.g:
`PUT /namespaces/{namespace}/tables/{table}/rename`
`PUT /namespaces/{namespace}/tables/{table}/move`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765387171
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
Review comment:
Is this meant for `UpdateProperties` request or is this a zombie comment coming from somewhere else?
I ask because at the time you wrote this, `UpdatePropertiesRequest` didn't exist and there's been some large reordering and github has had some trouble keeping up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765369860
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
So since we have `type`, which can be used to enumerate all of the possible sub fields, we've moved in the direction of just using the 3-digit HTTP response code in the body.
This file isn't part of the spec at all, as mentioned below, so I forgot to update it. I will remove it for now actually as it was just to show somebody something.
Do you input on using dedicated `code` (perhaps not constrained to HTTP code space as you mentioned) instead of just using the standardized value in `type` (which is a string)?
If we used `code`, the lookup table in my experience brings you to `type`. But I have previously used the way you described, with numeric error codes that we define a lookup table for that are independent. Do you have feedback on moving to just `type`?
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
+ */
+@JsonInclude(JsonInclude.Include.NON_NULL)
+public class IcebergHttpResponse<T> {
+
+ private final T data;
+ private final Error error;
+
+ @JsonCreator
+ public IcebergHttpResponse(
+ @JsonProperty("data") T data,
+ @JsonProperty("error") Error error) {
+ this.data = data;
+ this.error = error;
+ }
+
+ public Error error() {
+ return error;
+ }
+
+ public T data() {
+ return data;
+ }
+
+ /**
+ * An error object embedded in every HTTP response.
+ *
+ * On error, this contains:
+ * - message: A short, human-readable description of the error.
+ * - type: Type of exception - more specifically a class name, e.g. NamespaceNotFoundException)
Review comment:
No this isn't part of the spec at all.
I uploaded this here to share it with somebody I was speaking to, but this file is not part of the spec whatsoever. Sorry about the confusion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766110029
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
Review comment:
> I'd suggest using the terms normative and optional instead. It is not clear what the "overrides" are overriding since the intention is obviously for these values to be the source of truth.
For me, `normative` is rather confusing. The word `overrides` was used meaning that it would override any client supplied value as it's the source of truth (e.g. it's required).
I'm open to using a different word, but I haven't typically heard `normative` used in this sense and I think it would cause more confusion for end users than choosing some other word.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766209799
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
Review comment:
At this point, no. Not that I'm aware of given that we have the above for people to make test calls via Swagger UI.
At some point, with the way that some of the libraries work, it might become necessary to add something like this (I needed it for initially trying unit testing via this file).
I'm happy to remove it for now :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dimas-b commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
dimas-b commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766032201
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
IMHO, `type` is preferable to numeric `code`, but it might be harder to define strictly and there's risk that `type` names deviate from actual exception class names. Plus fixing class names in the API spec might affect non-java implementations in a negative way.
In Nessie, as an example, we have an enum of non-numeric error codes. The API returns enum names, but they can be mapped to java exceptions upon parsing. Python client has its own mapping too. cf. [ErrorCode.java](https://github.com/projectnessie/nessie/blob/main/model/src/main/java/org/projectnessie/error/ErrorCode.java)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766206721
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
Ok. Then we'll stick with `type` for now and use the http response codes for `code`.
For the moment, there's no hard requirement that they map to exception names. And this class isn't actually part of the spec, it was more to show an idea to somebody.
We don't have to use class names / exception names. For now, let's just say that they are a set of well-defined names (at least for any values that we might want the client to react to).
Though implementations could in theory use a different set of names and they might have benefit for logging purposes / monitoring purposes, but the client wouldn't be able react to any custom names (because it wouldn't know about them and how to react to them). We don't ideally want to restrict the spec so much that people can't make it suit their needs (for example, if having a richer set of `type` would help them with observability in the future).
We can cover what those values are called in a separate discussion later on (this is far from the only discussion / reviewable that will be had about this so I think it's safe to punt it down the road, with that knowledge in mind that that's the general goal for now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765386600
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
Review comment:
Well, exception classes from inside of Iceberg was what I meant when I wrote that. In order for the catalog client to make use of them, they do need to be standardized.
A separate field where people can add more a more specific internal type if they wanted - that the client might not use (except generically - e.g. adding to spans or eventual metrics or something).
But "exception message" was a poor choice of words on my part. That was just meant as an example. I'll remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765390241
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
Review comment:
That's a great idea.
It could also be used as different query parameters (like a `strategy`), since such a concept doesn't currently exist. For this initial PR, I'm trying to stick to reall to just the things needed for addressing namespaces, while leaving things open for evolving.
But an enum could make sense as well. Let me follow up on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] laurentgo commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
laurentgo commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766763554
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+
+ #############################
+ # Reusable Response Objects #
+ #############################
+ responses:
+
+ IcebergResponseObject:
Review comment:
In my own experience, having the actual body as the top level object is not really an issue in Java/Javascript. It should also simplify a bit your OpenAPI definitions? Currently ResponseDataObjects is defined as an object, but for each operation it would be better to have something more defined, and I'm not sure how easy it is to add extra constraints on the `data` field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] laurentgo commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
laurentgo commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766749785
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
Review comment:
I guess it's more a suggestion than a specification? But I took it from https://swagger.io/specification/#operation-object
> Unique string used to identify the operation. The id MUST be unique among all operations described in the API. The operationId value is case-sensitive. Tools and libraries MAY use the operationId to uniquely identify an operation, therefore, it is RECOMMENDED to follow common programming naming conventions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751522591
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
why do we need a specific path for properties? It could just be a part of the namespace response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751600521
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
+ parameters:
+ - name: namespace
+ in: query
+ description: Namespace under which to list namespaces. Leave empty to list all namespaces in the catalog
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List all namespaces, or all namespaces underneat a given namespace
+ description: List namespaces underneath a given namespace
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ get:
+ tags:
+ - Catalog API
+ summary: Get the configured properties of a namespace
+ operationId: getNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ 'Application/JSON':
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
+ type: object
+ properties:
+ currentSnapshotTo:
+ $ref: '#/components/schemas/TableMetadata'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /v1
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+ NoSuchNamespaceError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request does not exist
+ NamespaceAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request already exists
+ NoSuchTableError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: The given table identifier does not exist
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+ IcebergResponseObject:
+ type: object
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ properties:
+ data:
+ $ref: '#/components/schemas/ResponseDataObject'
+ error:
+ $ref: '#/components/schemas/ResponseErrorObject'
+ example: "{ data: { }, error: { } }"
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body.
+ example: '{ data: { identifiers: [ "office.employees", "office.dogs", "office.cats" ] }'
+ ResponseErrorObject:
+ type: object
+ description: Error portion embedded in all JSON responses with further details of the error
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Machine-type of the errror, such as an exception class
+ code:
+ type: integer
+ description: Application specific error code, to disambiguage amongst subclasses of HTTP responses
+ example: 40101
+ metadata:
+ type: object
+ description: Additional metadata to accompany this error, such as server side stack traces or user instructions
+ nullable: true
+ required:
+ - message
+ - type
+ - code
+ examples:
+ catalogName:
+ value: prod
+ namespace:
+ value: prod.accounting
+ no_such_table_error:
+ value: |-
+ {
+ message: "The given table does not exist",
+ type: "NoSuchTableException",
+ code: 40101
+ }
+
+ no_such_iceberg_table_eror:
+ value: |-
+ {
+ message: "The given table does not exist",
Review comment:
Although this is an example, it should probably have `The given table is not an iceberg table` and use type `NoSuchIcebergTableException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751583067
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
Review comment:
I updated the `description` on line 28 in the top level `info` block, and bumped the `version` to 1.0.0 to clarify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r768124738
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
My concern with this is that what you're calling `move` and `rename` are currently just one action. So i think it would be possibly misleading to have them as two separate things, when they are really one thing (at least in the current API).
The rename table doesn't move any of the data presently. So what would happen if somebody `PUT ...action=move` and that worked, and then `PUT ....action=rename` and that call failed? What state would the system be in? Or what state would the system be in even after just a `move` call?
As for the `/actions` idea, I personally do like that and wouldn't be opposed.
I'll keep it as is for now so this discussion doesn't get moved / erased, and then we can talk about the `/actions` thing coming up.
This is still very much a work in progress. So here would sitll be time for later changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-997110458
Closed in favor of https://github.com/apache/iceberg/pull/3770/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754722445
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
Review comment:
Do we need pagination?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754719210
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
Review comment:
`A table name`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757708356
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
Review comment:
So this `additionalProperties` is something that came up often for showing types in the object or that the object generally has some schema-less portion.
However, I think we can get rid of it and the generated document is still sufficient. It doesn't seem to be a widely enough recognized configuration property in these docs to be beneficial. I'll try to remove it if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757700351
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
Review comment:
But yes that does need some sort of response code in it for namespace doesn't exist. Will add that. So far I've been using `412 - Precondition Failed` as namespace existing seem somewhat like a precondition that has failed.
I don't love it bu that's the rationale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757702618
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
Review comment:
I found some of a URI standard [here](https://tools.ietf.org/html/rfc3986), though standards are often times widely not implemented in some parts so I'd rather go with what's common over anything else.
From Section 2.3:
> URIs that differ in the replacement of an unreserved character with
> its corresponding percent-encoded US-ASCII octet are equivalent: they
> identify the same resource. However, URI comparison implementations
> do not always perform normalization prior to comparison (see Section
> 6). For consistency, percent-encoded octets in the ranges of ALPHA
> (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E),
> underscore (%5F), or tilde (%7E) should not be created by URI
> producers and, when found in a URI, should be decoded to their
> corresponding unreserved characters by URI normalizers.
>
From Section 2.4:
> When a URI is dereferenced, the components and subcomponents
> significant to the scheme-specific dereferencing process (if any)
> must be parsed and separated before the percent-encoded octets within
> those components can be safely decoded, as otherwise the data may be
> mistaken for component delimiters. The only exception is for
> percent-encoded octets corresponding to characters in the unreserved
> set, which can be decoded at any time.
So it seems that the suggested approach is to slice the URL and then normalize each part if necessary.
To me, that makes it seem like one `namespace` path component that it is itself URL encoded if needed (to handle dots etc) would be preferred. But that's just based on my reading of a standard. I'd also consider putting `namespace` as a query parameter, for example when listing, but would prefer not to make the endpoints too different w.r.t. parameters.
I've mostly I'd love to hear any alternative views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-980442740
Thanks for the reviews and questions everybody! It's very much appreciated.
It is a holiday in the US, but I will get to everybody's questions or input as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757700351
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
Review comment:
But yes that does need some sort of response code in it for namespace doesn't exist. Will add that. So far I've been using `412 - Precondition Failed` as namespace existing seem somewhat like a precondition that has failed.
I don't love it but that's the rationale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757714172
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
Review comment:
It seems that there is currently consensus around not tackling things that create a Transaction.
To make it easier to review this in parts, I'll remove anything that returns `Transaction` from this PR. That's these functions in the `Catalog` interface: https://github.com/apache/iceberg/blob/7bfecd8514b943b00415fc4f49d1c7626fe62f4c/api/src/main/java/org/apache/iceberg/catalog/Catalog.java#L129-L286
Dropping a table isn't technically a `Transaction`, but I do understand what you're saying. Dropping is a large table operation definitely.
I'm open to moving parts alongside the `Transaction` API into their own PR as well to make the initial discussion easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
flyrain commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r753583166
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
+ parameters:
+ - name: namespace
+ in: query
+ description: Namespace under which to list namespaces. Leave empty to list all namespaces in the catalog
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List all namespaces, or all namespaces underneat a given namespace
+ description: List namespaces underneath a given namespace
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ get:
+ tags:
+ - Catalog API
+ summary: Get the configured properties of a namespace
+ operationId: getNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ 'Application/JSON':
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
Review comment:
Does the rest server have to open the metadata.json file to get these?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754729545
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
Review comment:
Maybe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754728793
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
Yeah, we should make this `GET /namespaces/{namespace}`. We can probably still use the `loadNamespaceMetadata` name. I like using the names that correspond to the interfaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754735365
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
Review comment:
Can this be a `TableMetadata` type? I'd rather not pass JSON through a JSON string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754734011
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
Review comment:
I'd probably remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760550078
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
Review comment:
Yeah the example code is 5 digits.
The first 3 digits are the HTTP response code, the second two digits are defined by us.
In this example, the code is 41202: 412 being the HTTP response code, and 02 meaning "table does not exist". My thinking was just that we'd have a set of codes in the spec that we follow that one could easily look up to get more information on the cause of the failure.
I'm open to removing that field entirely though if we'd like (or just sticking the HTTP response code in the error body).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760573032
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
Review comment:
Is `/<version>/<api>` URI a standard in OpenAPI to separate different API versions? You will also have minor version updates of the API, it does not seem extensible to me. When I think about loading the catalog through endpoint, the endpoint itself could be `https://my.catalog.com/v1`, and the spec itself should not have such version information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760548757
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
That works for me. I'm happy to move this one back to `GET /namespaces/{namespace}/properties` and then leave the `POST` and update the other to `DELETE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760556310
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
Review comment:
How does `POST /tables/renameTable` sound? I think keeping `tables` in the path is a bit more clear than plain `rename`.
I'll do it that way. Let me know if it should be changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760548757
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
That works for me. I'm happy to move this one back to `GET /namespaces/{namespace}/properties` and then leave the `POST` and `PUT`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760580041
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
Review comment:
I think @rymurr is asking why we include the fixed parts of the path, `namespaces` and `tables`.
That's because we want to avoid conflicts with routes like `/v1/namespaces/{ns}/properties`. We want to be able to have that route for properties, but also support tables named properties at `/v1/namespaces/{ns}/table/properties`.
Another option is to use different prefixes, like `/properties/{ns}`. But then you end up with a lot of prefixes, and I think that those tend to lack the helpful context. What does `/properties/accounting/q3_2021` return? Information about a table or a namespace? It isn't obvious from the path, but a slightly more verbose `/namespaces/accounting/tables/q3_2021/properties` is clear. (Of course, you can add even more prefixes, like `/namespace_properties/{ns}` and `/table_properties/{ns}/{tab}`)
I like the approach of including the context and having the hierarchy there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760597114
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
Review comment:
Per @rdblue's request, I moved it back from `PUT /tables/{table}` to an explicit `POST /v1/tables/renameTable`.
This way, both table identifiers are in the request body and there isn't any ambiguity about which table in the route corresponds to the source name or target name, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
flyrain commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r753579084
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
Review comment:
As a Catalog APIs, it probably won't touch underlying data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754587340
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
Review comment:
Removed `prod` from all of the example namespaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754730229
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
Review comment:
Can we specify string values in that object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754727421
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
Review comment:
This is actually not an overwrite operation. The `set` operation sets just the given properties and leaves the rest unchanged.
You might consider copying the documentation from the `SupportsNamespaces` and `Catalog` interfaces. for `setProperties`, we have "Properties that are not in the given map are not modified or removed by this method."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754728793
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
Yeah, we should make this `GET /namespaces/{namespace}`. We can probably still use the `loadNamespaceMetadata` name. I like using the names that correspond to the interfaces.
(+1 for removing `GET /namespaces/{namespace}/properties`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754726559
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
Review comment:
Does it make sense to have a namespace in the path for this? Both identifiers in the request are full identifiers. I think it would be more straightforward to have a `POST /rename` route instead of `PUT /namespaces/{namespace}/tables`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754574563
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
Review comment:
Yeah I'm open to other codes. I'm also open to keeping 404, but I have known it to cause monitoring issues (for example, monitoring just ELB response codes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754725122
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
Review comment:
Same here.
This should also state that it is valid to request moves between namespaces, but the implementation may not support it. If moving across namespaces is not allowed, then how about 406 Not acceptable for UnsupportedOperationException?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754734475
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
Review comment:
`tableToken`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754730970
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
Review comment:
Agreed. 412 for NoSuchNamespaceException? And probably 409 for NamespaceNotEmptyException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754453079
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
+ parameters:
+ - name: namespace
+ in: query
+ description: Namespace under which to list namespaces. Leave empty to list all namespaces in the catalog
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List all namespaces, or all namespaces underneat a given namespace
+ description: List namespaces underneath a given namespace
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ get:
+ tags:
+ - Catalog API
+ summary: Get the configured properties of a namespace
+ operationId: getNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ 'Application/JSON':
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
+ type: object
+ properties:
+ currentSnapshotTo:
+ $ref: '#/components/schemas/TableMetadata'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /v1
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+ NoSuchNamespaceError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request does not exist
+ NamespaceAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request already exists
+ NoSuchTableError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: The given table identifier does not exist
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+ IcebergResponseObject:
+ type: object
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ properties:
+ data:
+ $ref: '#/components/schemas/ResponseDataObject'
+ error:
+ $ref: '#/components/schemas/ResponseErrorObject'
+ example: "{ data: { }, error: { } }"
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body.
+ example: '{ data: { identifiers: [ "office.employees", "office.dogs", "office.cats" ] }'
+ ResponseErrorObject:
+ type: object
+ description: Error portion embedded in all JSON responses with further details of the error
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Machine-type of the errror, such as an exception class
+ code:
+ type: integer
+ description: Application specific error code, to disambiguage amongst subclasses of HTTP responses
+ example: 40101
Review comment:
This looks incorrect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754617447
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
Review comment:
Ah ok, will update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754735890
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
+ type: object
+ properties:
+ currentSnapshotTo:
+ $ref: '#/components/schemas/TableMetadata'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
Review comment:
We should consider whether to make this `prefix`. I tend to prefer simpler names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754735757
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
+ type: object
+ properties:
+ currentSnapshotTo:
Review comment:
This looks strange to me. Is this part ready for review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754735204
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
Review comment:
I'd probably remove everything but the identifier, metadataLocation, and metadataJson. We don't need to send everything twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757701530
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
Review comment:
I've URL encoded some examples with dots and with other url characters and my validation system said they were valid.
Do you know of any restrictions on URL encoding path variables that some load balancers or frameworks / common libraries won't allow url encoded values as path variables?
I have been considering potential alternative options so I appreciate the input. :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757707515
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
Review comment:
So one things I should emphasize is that the entire markdown file was autogenerated from the YAML.
That said, I did put the base url and port into the open api document, more just for testing purposes (with use with a mock server library to be specific).
I can look into how to make them configurable in an OpenAPI document and then would be happy to do that.
We can also add commands or instructions on how to generate the markdown oneself if they'd like (or maybe the generator tool I'm using will respect the user's entered host and port in the markdown if that's an option in an OpenAPI document).
If you happen to know how off the top of your head or have a recommended way to add those as config options to the OpenAPI doc, by all means I'd love to hear it. I'll also look to see if there's a standard way to do that as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
nastra commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763083367
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 40401 }'
Review comment:
40401 -> 404?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 40401 }'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
Review comment:
I believe a `HEAD` request must not return any content. You could indicate that the resource exists via HTTP status code 204 and use 404 if it doesn't exist
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 40401 }'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ example: '{ "data": { "exists": false } }'
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ example: '{ "data": { "dropped": true } }'
Review comment:
will this return `dropped = false` when calling this endpoint again after the resource has been deleted?
I think this endpoint is also missing a 404 status code if the resource doesn't exist, unless we want to return dropped=true/false depending on whether the resource was deleted or not.
`DELETE` is supposed to be idempotent, but I think it's ok to actually return dropped=true/false as long as the resource was deleted on the first call and additional calls of this endpoint won't change the state on the server, but return dropped=false
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
Review comment:
code should probably be 409 instead of 40901
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 40401 }'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ example: '{ "data": { "exists": false } }'
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
Review comment:
`must be empty` -> `...must not be empty` or `... must be non-empty`
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
Review comment:
+1 to either use POST or PUT. Also it seems weird to use PUT to actually delete properties. PUT is supposed to be idempotent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763122554
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 40401 }'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ example: '{ "data": { "exists": false } }'
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
Review comment:
Oh here's the [PR](https://github.com/apache/iceberg/pull/3643) that fixes the really glaring typo in that doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763122111
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 40401 }'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ example: '{ "data": { "exists": false } }'
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
Review comment:
I think the requirement (per the existing `dropNamespace` API) is that it must be empty.
https://github.com/apache/iceberg/blob/7fcc71da65a47ca3c9f6eb6e862a238389b8bdc5/api/src/main/java/org/apache/iceberg/catalog/SupportsNamespaces.java#L103-L105
I've opened a PR to fix the typo there, but that's where I got that from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763251503
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
Review comment:
Added 422 (Unprocessable Entity) to indicate that there was a key in both toRemove and toUpdate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] laurentgo commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
laurentgo commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r764644950
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
Review comment:
Looking at the semantic of the API, maybe a body is not necessary and can be substituted with 2 different status:
- `204: No Content` is DELETE was successful
- `409: Conflict` if the DELETE was not done because the object was not empty
It is also clear based on the status code if the operation was successful or not...
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
Review comment:
Typo: `Namespace exists`
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
Review comment:
should an exception class be used? wouldn't that be implementation specific?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
Review comment:
(nit) maybe avoid boolean and use some enum value so that we may be able to extend with different purge policy?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
Review comment:
For the sake of interoperability, wouldn't it be better to have something more prescriptive than just a dictionary of properties?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
Review comment:
typo: `must return Namespace.of("a", "b")`
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
Review comment:
Why not using PATCH on path `/namespaces/{namespace}` since that method would correspond to that definition (note that there are also rfc proposal for json patch format: https://datatracker.ietf.org/doc/html/rfc6902 and https://datatracker.ietf.org/doc/html/rfc7386)
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
Review comment:
Is there a way to detect/prevent concurrent modifications if needed?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
Review comment:
I hope this is not an actual set of credentials? I'm kind of concern about a server config endpoint returning a security header tbh...
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
Why not keeping a RESTFul approach and make it a method of the original table?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
Review comment:
since 200 is returned, does it mean there should be a response body in case of success too?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
Review comment:
I thought it was recommended for operationId to be shaped like program identifiers (which would prevent the use of space and slash?)
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
Review comment:
do we expect http status codes below 400?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+
+ #############################
+ # Reusable Response Objects #
+ #############################
+ responses:
+
+ IcebergResponseObject:
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ content:
+ application/json:
+ schema:
+ anyOf:
+ - $ref: '#/components/schemas/ResponseDataObject'
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ example: { "data": { "namespaces": [ [ "ns1", "foo_db" ], [ "ns1", "bar_db" ] ] }, "error": { } }
Review comment:
should it be `anyOf` or `oneOf`? Wouldn't that make it even more confusing if the response has both a data and an error field?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
Review comment:
typo: beneficial
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+
+ #############################
+ # Reusable Response Objects #
+ #############################
+ responses:
+
+ IcebergResponseObject:
Review comment:
Wrapping a successful response into another `data` field is not something I have seen many APIs doing. Convention seems to be that the true object to be at the top level? Is the motivation for that choice to be able to quickly identify if the response is an error message from a successful one?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
Review comment:
Sadly I don't think there's a way yet to have common responses across all operations. See https://swagger.io/docs/specification/describing-responses/ section: `Reusing responses`
Also, 400 is probably not the right code for "Unknown error" but for Bad Request (as in the client submitted some bad request). I would assume this is a 500 error code...
I'm also guessing that if 401 is returned, 403 (Forbidden) may be returned as well if the server denies caller access based on its permission level?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769890854
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+
+ #############################
+ # Reusable Response Objects #
+ #############################
+ responses:
+
+ IcebergResponseObject:
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ content:
+ application/json:
+ schema:
+ anyOf:
+ - $ref: '#/components/schemas/ResponseDataObject'
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ example: { "data": { "namespaces": [ [ "ns1", "foo_db" ], [ "ns1", "bar_db" ] ] }, "error": { } }
Review comment:
I am open to better ways to format the OpenAPI doc to express this. We have agreed on using `error` at least, but if there's a better way in OpenAPI to define it, I'd be happy to hear it or possibly a link to a description or example would help. 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769887154
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
Review comment:
I removed reference to `exception class` from the description.
I chose to leave the values as they currently are, as we don't have enough functionality defined to enumerate all possible values needed so far and that would also add much more overhead to reviewing this 😅 So I figure we can defer defining the values that will be "well-defined" to a later time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r768120704
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
Review comment:
Thinking on it further, I do really like the idea, but I think that for now we're trying to keep this to just the things we know we need. I introduced a few other additional things that were turned down under the You Ain't Gonna Need It theory, which I can respect.
If enough people disagree (or enough people think we should make it an enum), I'm open to that too. Just clarifying that previous discussion so far has mostly followed the principal of only what is needed for the most part.
We could also evolve the API over time, such that if a `strategy` were added, then the `purgeRequested` would be implicitly true not provided and a strategy that indicated `purgeRequested` was provided. Or just following the semantics of the `strategy` if / when that's needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r768103314
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
Review comment:
It could arguably be 201. I kind of prefer 200 as we don't necessarily need to have the body returned (at least the current API doesn't require it to be returned). With the standardized response body (or always returning at least some JSON in any response that allows for a body), we could still add it later if we found need for it.
We could always switch to 201 if we find that to be more appropriate as this evolves.
From the link you sent:
```
The new resource is effectively created before this response is sent back and the new resource is returned in the body of the message, its location being either the URL of the request, or the content of the Location header.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766206721
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
Ok. Then we'll stick with `type` for now.
For the moment, there's no hard requirement that they map to exception names. And this class isn't actually part of the spec, it was more to show an idea to somebody.
We don't have to use class names. For now, let's just say that they are a set of well-defined names (at least for any values that we might want the client to react to).
Though implementations could in theory use a different set of names for logging purposes / monitoring purposes, but the client wouldn't be able react to any custom names (because it wouldn't know about them and how to react to them).
We can cover what those values are called in a separate discussion later on (this is far from the only discussion / reviewable that will be had about this so I think it's safe to punt it down the road, with that knowledge in mind that that's the general goal for now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751493263
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
Review comment:
Good call. Consistency is good. I'm definitely open to updating the routes as needed for consistency and to avoid any potential issues with generating the requests.
I also have a few more examples of the current JSON that gets generated, but if you'd like to sync up on that as JSON representation is generally needed for the Trino side, let me know. 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754636277
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
Review comment:
Oh I like that much more. Avoiding the use of `namespace` as a query parameter when it's a path parameter everywhere else is likely to be a lot less confusing.
I'll update the route and the description to clarify. Thanks Jack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754636778
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
Review comment:
Dropped. Will do a pass over to make sure I don't have much of these left in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754719066
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
Review comment:
Nit: This presumes the existence of a table. I would simply say "A namespace identifier".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754717875
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
Review comment:
This states that the settings are overrides. But what about defaults? Should we have a way to specify whether a setting _must_ be used or is just a default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754619476
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
I chose to remove the operation that was `getNamespace`, since its essentially the same response and there is not an interface for just getting a namespace (just `loadNamespaceMetadata`).
We do need some sort of `properties` route for `setProperties` and `removeProperties`, hence why I left it here.
Let me know if you have any concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754734959
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
Review comment:
Is this within `properties`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754580217
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
Review comment:
The use case would be storing catalog specific properties, like maybe the FileIO implementation that's to be used. I'll drop it for now though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754730067
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
Review comment:
`namespaces`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754729282
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
Review comment:
Looks like a copy & paste problem. Why have `NoSuchNamespaceException` and `NamespaceAlreadyExists` exception? Either it should or shouldn't exist before using this route. (It should.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754725738
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
Review comment:
What about `fromIdentifier` and `toIdentifier`? I think it's clear that these are tables, and `from` and `to` are easier to understand than source and target.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757707621
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
+ type: object
+ properties:
+ currentSnapshotTo:
+ $ref: '#/components/schemas/TableMetadata'
+ IcebergConfiguration:
Review comment:
Yeah I prefer that. I don't like having `Iceberg` in the configuration names. Good suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757714453
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
Review comment:
Not necessarily.
Some people might need pagination, but it's not really part of the current API. i can remove the pagination from here and elsewhere if we'd like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] laurentgo commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
laurentgo commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r755700012
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
Review comment:
IMHO this is kind of an abuse of 412. The RESTful convention is that 404 is for non-existing resources. I'm not sure why we should depart from it to make it easier for internal monitoring (shouldn't the load balancer be configured with the routes and could have separate metrics regarding non-existent route vs the backend returned 404 for example)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751521507
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
Review comment:
Does this v1 mean the Iceberg v1 spec, or the v1 of this Rest API definition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dimas-b commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
dimas-b commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r764440889
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
Review comment:
nit: is an empty `error` object meaningful? Should it be `null` in this case, perhaps?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
Review comment:
I'd suggest using the terms `normative` and `optional` instead. It is not clear what the "overrides" are overriding since the intention is obviously for these values to be the source of truth.
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
Could it be clearer if the HTTP error code were separate from the internal error code? This would allow free form internal code without having to be confined to 100 values under an HTTP code prefix, and at the same time maintain reference to the more generic HTTP set of codes. It would also make it easier to extract HTTP error codes for processing.
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
Review comment:
By the way, what is a use case for the "default" properties? Could you add an example, please?
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
+ */
+@JsonInclude(JsonInclude.Include.NON_NULL)
+public class IcebergHttpResponse<T> {
+
+ private final T data;
+ private final Error error;
+
+ @JsonCreator
+ public IcebergHttpResponse(
+ @JsonProperty("data") T data,
+ @JsonProperty("error") Error error) {
+ this.data = data;
+ this.error = error;
+ }
+
+ public Error error() {
+ return error;
+ }
+
+ public T data() {
+ return data;
+ }
+
+ /**
+ * An error object embedded in every HTTP response.
+ *
+ * On error, this contains:
+ * - message: A short, human-readable description of the error.
+ * - type: Type of exception - more specifically a class name, e.g. NamespaceNotFoundException)
Review comment:
Does this have to be part of the spec? In other words are specific exception class names going to be part of the spec in addition and/or in parallel to "internal" error codes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] omarsmak commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
omarsmak commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r764661610
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
Review comment:
Shouldn't this be [`201`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/201) instead?
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
I am concern here to mix to functionalities into one method. Why don't we just separate these two functionalities and move under the table endpoint? E.g:
`PUT /namespaces/{namespace}/tables/{table}/rename`
`PUT /namespaces/{namespace}/tables/{table}/move`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765369860
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
So since we have `type`, which can be used to enumerate all of the possible sub fields, we've moved in the direction of just using the 3-digit HTTP response code in the body.
This file isn't part of the spec at all, as mentioned below, so I forgot to update it. I will remove it eventually but leave it so we can discuss.
Do you have input on using a dedicated int `code` (perhaps not constrained to HTTP code space as you mentioned) instead of just using the standardized value in `type` (which is a string)?
If we used `code`, the lookup table in my experience brings you to `type`. But I have previously used the way you described, with numeric error codes that we define a lookup table for that are independent. Do you have feedback on moving to just `type`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765384883
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
Review comment:
It really is just a map of properties though, at least as far as the spec considers it.
We could possibly add some specific properties onto it, but we'd have to allow for general string key-value map entries still.
What entries would you have in mind? I don't believe there are any standard properties for a namespace. For table's and for catalog's yes, but not for namespaces as far as I'm aware.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766092890
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
But with your proposed `/rename` and then `/move` route, it would be supportable once that exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766101374
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
Review comment:
I had not heard that before. Do you have any source indicating that?
I updated this to use the slash because this one route is handling both `setPrpoperties` call and `removeProperties` call: https://github.com/apache/iceberg/blob/59a76da2ad78b3fabed024baa359e931d239ddc9/api/src/main/java/org/apache/iceberg/catalog/SupportsNamespaces.java#L107-L129
editor.swagger.io did not complain about this operationId (and it seems to complain about a number of other things), but if what you're is true, then probably the operationId could be `setOrRemoveProperties` or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769859433
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
> Re: type values, I personally tend to think that if the field itself is part of the spec, the spec would have to define how to interpret it. I am not sure keeping it "opaque" is going to be that much useful from the spec's perspective.
I agree. Since we're using code to just be the HTTP status response code, I would say that `type` should be a well defined set we can look up.
We might also choose to allow people to use others that will get ignored by the client but are logged or something.
But for now I would say let's plan for it to be a well defined set of values, but not necessarily attempt to enumerate all of them just yet since we don't have about half of the basic Catalog AP in here yet. It's currently mostly just around namespaces and some of the table related ones will be handled next.
Sound good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760680578
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
Review comment:
I'll add 202 to the response as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760683101
##########
File path: rest_docs/rest-catalog-v0.1-SNAPSHOT.md
##########
@@ -0,0 +1,1810 @@
+---
+title: Apache Iceberg REST Catalog API v1.0.0
+language_tabs:
+ - shell: Shell
+ - java: Java
+language_clients:
+ - shell: ""
+ - java: ""
+toc_footers: []
+includes: []
+search: true
+highlight_theme: darkula
+headingLevel: 2
+
+---
+
+<!-- Generator: Widdershins v4.0.1 -->
+
+<h1 id="apache-iceberg-rest-catalog-api">Apache Iceberg REST Catalog API v1.0.0</h1>
+
+> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
+
+Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+
+Base URLs:
+
+* <a href="http://127.0.0.1:1080">http://127.0.0.1:1080</a>
+
+License: <a href="https://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0</a>
+
+# Authentication
+
+- HTTP Authentication, scheme: bearer
+
+<h1 id="apache-iceberg-rest-catalog-api-configuration-api">Configuration API</h1>
+
+## getConfig
+
+<a id="opIdgetConfig"></a>
+
+> Code samples
+
+```shell
+# You can also use wget
+curl -X GET http://127.0.0.1:1080/v1/config \
+ -H 'Accept: application/json' \
+ -H 'Authorization: Bearer {access-token}'
+
+```
+
+```java
+URL obj = new URL("http://127.0.0.1:1080/v1/config");
+HttpURLConnection con = (HttpURLConnection) obj.openConnection();
+con.setRequestMethod("GET");
+int responseCode = con.getResponseCode();
+BufferedReader in = new BufferedReader(
+ new InputStreamReader(con.getInputStream()));
+String inputLine;
+StringBuffer response = new StringBuffer();
+while ((inputLine = in.readLine()) != null) {
+ response.append(inputLine);
+}
+in.close();
+System.out.println(response.toString());
+
+```
+
+`GET /v1/config`
+
+*List all catalog configuration settings*
+
+All REST catalogs will be initialized by calling this route. This route will return at least the minimum necessary metadata to initialize the catalog. Optionally, it can also include server-side specific overrides. For example, it might also include information used to initialize this catalog such as the details of the Http connection pooling, etc. This route might also advertise information about operations that are not implemented so that the catalog can eagerly throw or go about another way of performing the desired action.
Review comment:
I will give some thought as to how this could look and add an example. This seems reasonable to me and I prefer it over my suggestion of having each key specify if it's an override or a default that the client can choose to reject.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760567317
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
Review comment:
Will figure out a place to move it so it's just one comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
nastra commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763148876
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,586 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+# TODO - Add ServerVariable so user's can control the routes and ports etc and have mutli-env etc.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ description: Conflict (AlreadyExistsException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 40401 }'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ example: '{ "data": { "exists": false } }'
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ example: '{ "data": { "dropped": true } }'
Review comment:
I think this specific use case when deleting something isn't super well defined in REST. One could argue that the cleanest approach would be to perform the DELETE when the resource exists and error out with a 404 if it doesn't exist.
On the other hand you could still model it the same way as you did. The outcome on the server is the same, the resource is deleted (hence the endpoint is idempotent). The only difference is how you model the case when the resource doesn't exist (anymore)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r763406350
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
+ parameters:
+ - name: namespace
+ in: query
+ description: Namespace under which to list namespaces. Leave empty to list all namespaces in the catalog
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List all namespaces, or all namespaces underneat a given namespace
+ description: List namespaces underneath a given namespace
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ get:
+ tags:
+ - Catalog API
+ summary: Get the configured properties of a namespace
+ operationId: getNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ 'Application/JSON':
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
+ type: object
+ properties:
+ currentSnapshotTo:
+ $ref: '#/components/schemas/TableMetadata'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /v1
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+ NoSuchNamespaceError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request does not exist
+ NamespaceAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request already exists
+ NoSuchTableError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: The given table identifier does not exist
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+ IcebergResponseObject:
+ type: object
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ properties:
+ data:
+ $ref: '#/components/schemas/ResponseDataObject'
+ error:
+ $ref: '#/components/schemas/ResponseErrorObject'
+ example: "{ data: { }, error: { } }"
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body.
+ example: '{ data: { identifiers: [ "office.employees", "office.dogs", "office.cats" ] }'
+ ResponseErrorObject:
+ type: object
+ description: Error portion embedded in all JSON responses with further details of the error
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Machine-type of the errror, such as an exception class
+ code:
+ type: integer
+ description: Application specific error code, to disambiguage amongst subclasses of HTTP responses
+ example: 40101
Review comment:
Reverted everything to 3 digit codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765381142
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
Oh. So using `PUT` with `/rename` might not be considered as overloading.
That's a good suggestion.
As for with `rename`vs`move`, can you clarify further which two functionalities you mean? What would `rename` do that `move` wouldn't, and similarly what would `move` do that `rename` wouldn't? Or when would each be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765370724
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
Review comment:
Yeah I've previously treated it as both. Usually I would serialize it as `null` (or actually simple forgo serializing it, as most things can handle an object with one of two fields).
I left it here as an example. But I can remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765388905
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
+ example: NoSuchNamespaceException
+ code:
+ type: integer
+ minimum: 100
+ maximum: 600
+ description: HTTP response code
+ example: 404
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ nullable: false
+ name:
+ type: string
+ nullable: false
+ CreateNamespaceRequest:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ description: Individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ "owner": "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ UpdatePropertiesRequest:
+ type: object
+ properties:
+ toRemove:
+ type: array
+ uniqueItems: true
+ items:
+ type: string
+ example: '[ "department", "access_group" ]'
+ toUpdate:
+ uniqueItems: true
+ type: object
+ items:
+ type: string
+ example: { "owner": "Hank Bendickson" }
+ UpdatePropertiesResponse:
+ type: object
+ description: JSON data response for a synchronous update properties request.
+ properties:
+ updated:
+ type: array
+ items:
+ type: string
+ description: List of property keys that were added or udpated
+ removed:
+ type: array
+ items:
+ type: string
+ description: List of properties that were removed (and not updated)
+ notPresent:
+ type: array
+ items:
+ type: string
+ description:
+ List of properties requested for removal that were not found
+ in the namespace's properties.
+ Represents a partial success response.
+ Server's do not need to implement this.
+ ListNamespacesResponse:
+ type: object
+ properties:
+ namespaces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ DropTableResponse:
+ type: object
+ properties:
+ dropped:
+ type: boolean
+ description: true if the table was found and removed from the catalog
+ purged:
+ type: boolean
+ description: whether the underlying data was purged or is being purged
+ Namespace:
+ type: array
+ description: Reference to one or more levels of a namespace
+ items:
+ type: string
+ example: [ "accounting", "tax" ]
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' }
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+
+ #############################
+ # Reusable Response Objects #
+ #############################
+ responses:
+
+ IcebergResponseObject:
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ content:
+ application/json:
+ schema:
+ anyOf:
+ - $ref: '#/components/schemas/ResponseDataObject'
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ example: { "data": { "namespaces": [ [ "ns1", "foo_db" ], [ "ns1", "bar_db" ] ] }, "error": { } }
Review comment:
I was thinking in the case of partial success and partial error (such as in a microbatch process of statements), which is where I've used that. Though that can usually be defined in the response body inside just the data-intended section.
So mostly likely it should be `oneOf`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dimas-b commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
dimas-b commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r767853990
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
Excluding this class from the spec and using only OpenAPI yaml SGTM :+1:
Re: `type` values, I personally tend to think that if the field itself is part of the spec, the spec would have to define how to interpret it. I am not sure keeping it "opaque" is going to be that much useful from the spec's perspective.
Is this spec intended to define just a common part of the protocol between a Catalog impl. on the Iceberg side and the HTTP server that offers the REST API? Are the client and server parts supposed to be interoperable or are they assumed to work knowing the exact nature and version of the other side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766106281
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
Review comment:
I believe, from other discussion (much of which is lost in this PR becoming rather large), that the idea was to return a response body where possible to be able to more easily add additional metadata without having to completely change the response (e.g. the response type will still be an object).
So I don't disagree that it can be gleaned from the response status code, but the overall consensus so far was to return a body where possible so that the response body can be updated in the future and not require fully changing the type of the response (e.g. even if this just returned `{}`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765381142
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
Oh. So using `PUT` with `/rename` might not be considered as overloading.
That's a good suggestion.
EDIT: I understand now what you mean. Will follow up about move and rename 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766217147
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
Review comment:
I don't see it as being too different from a `/login` endpoint tbh.
Here's what I found for some basic login information on the web: https://mobile-security.gitbook.io/mobile-security-testing-guide/general-mobile-app-testing-guide/0x04e-testing-authentication-and-session-management#testing-stateful-session-management-mstg-auth-2
- The app sends a request with the user's credentials to the backend server.
- The server verifies the credentials If the credentials are valid, the server creates a new session along with a random session ID.
- The server sends to the client a response that includes the session ID.
So this particular example of sending back the exact `Authorization` header might not be how it's done, but outside of the route being called `config`, it's not unheard of for a client to exchange its credentials for temporary session credentials (or otherwise), which it then uses for the lifetime of the app.
The above is more or less how I've done authentication on other applications.
It could in theory be moved to a dedicated route, but the idea is to allow for people to implement authentication according to their own measures. The specification for this one catalog isn't intended to be incredibly strict in how things are defined, so that it can fit several people's usages (e.g. people who use different forms of authentication for things).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769855907
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
Review comment:
We decided to add the `/v1` back in so that clients and servers know which version of the API they are working on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769877035
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
Review comment:
As for the complexity, I mean having all three of `POST` / `PUT` / `PATCH`, especially as `PATCH`.
There have been so far people in support of one or the other, but the consensus was to avoid putting too many things into one route, so as to make things a bit more self documenting. Particularly when `PATCH` could refer to any number of potential patches but so far it only corresponds to one.
Just for insight / clarity as to what has been discussed somewhat so far (both here ... somewhere in the PR discussion .... and elsewhere offline etc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751574869
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
Review comment:
I would be ok with that. I believe I was trying to balance not overloading the same route with too many verbs, but that's not necessarily an issue.
Let me see what I can do. 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r752496716
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
Review comment:
Updated to `catalogs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760602891
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
Review comment:
Since `dropTable` isn't part of the transaction API, I'm going to leave it here for now at least. https://github.com/apache/iceberg/blob/27642de089aa1f9844b927fc700b29e299d7ca81/api/src/main/java/org/apache/iceberg/catalog/Catalog.java#L313-L322
If other's feel strongly about it, I'm open to that discussion. But at least from the API point of view, it doesn't have the same complications that `Transaction` based APIs have. And it also isn't part of `TableOperations`.
The goal isn't to necessarily prescribe how the server-side handles this, just give them a place to call the API.
They could return `202 - Accepted` or something as well and have it be somewhat async. And existing catalogs are free to still implement `dropTable` in the way they presently do.
I think a discussion of moving it to `TableOperations` should maybe be a separate discussion (if we have it).
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
Review comment:
It seems that there is currently consensus around not tackling things that create a Transaction.
To make it easier to review this in parts, I'll remove anything that returns `Transaction` from this PR. That's these functions in the `Catalog` interface: https://github.com/apache/iceberg/blob/7bfecd8514b943b00415fc4f49d1c7626fe62f4c/api/src/main/java/org/apache/iceberg/catalog/Catalog.java#L129-L286
Dropping a table isn't technically a `Transaction`, but I do understand what you're saying. Dropping is a large table operation definitely.
I'm open to moving parts alongside the `Transaction` API into their own PR as well to make the initial discussion easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760562572
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
Review comment:
Added 406 `Not Acceptable`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760563954
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
Review comment:
I think this depends on whether we restructure the API around catalog and table like Ryan's suggestion. Let's come back to this after considering that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760579659
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
Review comment:
Yeah that's my bad. I'll update that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra edited a comment on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
nastra edited a comment on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-975539916
Have you thought about/considered something like https://github.com/eclipse/microprofile-open-api (just an example, but there are other libs available as well), which allows you to add OpenAPI annotations directly to your Java code, so that the OpenAPI spec+docs and the code are all at one place?
In combination with JAX-RS annotations, everything that defines a REST endpoint would be combined at a single place and can be easier consumed/reasoned about by developers, rather than having the Swagger definitions and the endpoint implementations across multiple files.
The `swagger.json|yaml` is then generated from the annotations, which we could then serve by a separate REST endpoint (it would basically show what you see when you upload the `swagger.yaml` to editor.swagger.io).
Below is a quick & short example that shows how this would look like in code:
```
@Path("config")
public interface ConfigurationApi {
@GET
@Produces(MediaType.APPLICATION_JSON)
@Operation(summary = "List all catalog configuration settings")
@APIResponses({
@APIResponse(
responseCode = "200",
description =
"All REST catalogs will be initialized by calling this route. This route\n"
+ "will return at least the minimum necessary metadata to initialize the\n"
+ "catalog. Optionally, it can also include server-side specific overrides.\n"
+ "For example, it might also include information used to initialize this catalog\n"
+ "such as the details of the Http connection pooling, etc. This route might\n"
+ "also advertise information about operations that are not implemented\n"
+ "so that the catalog can eagerly throw or go about another way of performing\n"
+ "the desired action.",
content =
@Content(
mediaType = MediaType.APPLICATION_JSON,
schema = @Schema(implementation = IcebergConfiguration.class))),
@APIResponse(responseCode = "400", description = "Unknown Error"),
@APIResponse(responseCode = "401", description = "Invalid credentials provided"),
})
IcebergConfiguration getConfig();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-975911260
> Have you thought about/considered something like https://github.com/eclipse/microprofile-open-api (just an example, but there are other libs available as well), which allows you to add OpenAPI annotations directly to your Java code, so that the OpenAPI spec+docs and the code are all at one place? In combination with JAX-RS annotations, everything that defines a REST endpoint would be combined at a single place and can be easier consumed/reasoned about by developers, rather than having the Swagger definitions and the endpoint implementations across multiple files. The `swagger.json|yaml` is then generated from the annotations, which we could then serve by a separate REST endpoint (it would basically show what you see when you upload the `swagger.yaml` to editor.swagger.io).
>
> Below is a quick & short example that shows how this would look like in code:
>
> ```
> @Path("config")
> public interface ConfigurationApi {
>
> @GET
> @Produces(MediaType.APPLICATION_JSON)
> @Operation(summary = "List all catalog configuration settings")
> @APIResponses({
> @APIResponse(
> responseCode = "200",
> description =
> "All REST catalogs will be initialized by calling this route. This route\n"
> + "will return at least the minimum necessary metadata to initialize the\n"
> + "catalog. Optionally, it can also include server-side specific overrides.\n"
> + "For example, it might also include information used to initialize this catalog\n"
> + "such as the details of the Http connection pooling, etc. This route might\n"
> + "also advertise information about operations that are not implemented\n"
> + "so that the catalog can eagerly throw or go about another way of performing\n"
> + "the desired action.",
> content =
> @Content(
> mediaType = MediaType.APPLICATION_JSON,
> schema = @Schema(implementation = IcebergConfiguration.class))),
> @APIResponse(responseCode = "400", description = "Unknown Error"),
> @APIResponse(responseCode = "401", description = "Invalid credentials provided"),
> })
> IcebergConfiguration getConfig();
> }
> ```
I looked into something like this. For the moment, I chose to go without it given that I wanted to get this presentable and wasn't exactly sure how to format existing things in this way. Also, I believe we have slightly different libraries already available for it, though I think those are coming form other modules.
This week is a holiday in the US (Thanksgiving) and I'm going to ry to play around with this annotation styled AP to see if I can generate the docs in this way, as I do prefer it.
Thanks as always for the pointer @nastra. Your example will help as typically I've seen this used with annotations that are more specific to Spring or some particular framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754732347
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
Review comment:
What's this pattern? No spaces? Should that be some identifierPart type that we use for namespaces as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754452307
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
Review comment:
I think using `prod` as the example will make people confuse this with catalogs. I would use something like `accounting` here and `accounting.tax` for the multipart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754572967
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
Review comment:
Oh you're right. I was thinking of the `listNamespaces()` call that lists all top level namespaces.
I will update this then as I much prefer to keep it consistent. My bad!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754573516
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
Review comment:
Happy to drop `purge`, but it is part of the Catalog interface: https://github.com/apache/iceberg/blob/f96f06ea07841c137a98a25e4edf62c508475801/api/src/main/java/org/apache/iceberg/catalog/Catalog.java#L287-L296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754600423
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
Review comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754729282
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
Review comment:
Looks like a copy & paste problem. Why have `NoSuchNamespaceException` and `NamespaceAlreadyExists` exception? Either it should or shouldn't exist before using this route.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754578590
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
Review comment:
Yeah that makes sense.
The doc comment on the API does somewhat imply that `purge` is somewhat optional, and there are a lot of use cases where somebody who implements things on their server side might choose to reject `purge` or ignore it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754727692
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
Review comment:
I'd probably opt to have either POST or PUT, but not both. It is more complicated to have both with slightly different behaviors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754736706
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
Review comment:
+1 for having `TableMetadata` that describes what is in the metadata JSON file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751469488
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
Review comment:
for consistency with namespace and table, I think this should be `/v1/catalogs/{catalog}`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754588219
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
Review comment:
So what I was trying to cover here was the `createTableTransaction` APIs, which are part of `Catalog`.
However, since the actual commit isn't part of `Catalog`, it's probably best to not include it yet since we're just covering `Catalog`. Will remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754603837
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
+ parameters:
+ - name: namespace
+ in: query
+ description: Namespace under which to list namespaces. Leave empty to list all namespaces in the catalog
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List all namespaces, or all namespaces underneat a given namespace
+ description: List namespaces underneath a given namespace
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ get:
+ tags:
+ - Catalog API
+ summary: Get the configured properties of a namespace
+ operationId: getNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ 'Application/JSON':
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ ListTablesResponse:
+ type: object
+ properties:
+ identifiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ GetTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ location:
+ type: string
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ TableMetadata:
+ type: object
+ properties:
+ currentSnapshotTo:
+ $ref: '#/components/schemas/TableMetadata'
+ IcebergConfiguration:
+ type: object
+ description: "Server-provided configuration for the catalog."
+ properties:
+ rootPath:
+ minLength: 1
+ type: string
+ default: /v1
+ nullable: false
+ description:
+ Root path to be used for all other requests. Server-side implementations
+ are free to use another choice for root path, but must conform to the
+ specification otherwise in order to interoperate with the URI generation
+ that the REST catalog client will do.
+ catalogProperties:
+ type: object
+ nullable: true
+ description:
+ An optional field for storing catalog configuration properties that
+ are stored server side. This could be beneifical to an administrator,
+ for example to enforce that a given LocationProvider is used or to
+ enforce that a certain FileIO implementation is used.
+ NoSuchNamespaceError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request does not exist
+ NamespaceAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: Namespace provided in the request already exists
+ NoSuchTableError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: The given table identifier does not exist
+ TableAlreadyExistsError:
+ allOf:
+ - $ref: '#/components/schemas/ResponseErrorObject'
+ description: the given table identifier already exists and cannot be created / renamed to
+ IcebergResponseObject:
+ type: object
+ description: JSON wrapper for all response bodies, with a data object and / or an error object
+ properties:
+ data:
+ $ref: '#/components/schemas/ResponseDataObject'
+ error:
+ $ref: '#/components/schemas/ResponseErrorObject'
+ example: "{ data: { }, error: { } }"
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body.
+ example: '{ data: { identifiers: [ "office.employees", "office.dogs", "office.cats" ] }'
+ ResponseErrorObject:
+ type: object
+ description: Error portion embedded in all JSON responses with further details of the error
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Machine-type of the errror, such as an exception class
+ code:
+ type: integer
+ description: Application specific error code, to disambiguage amongst subclasses of HTTP responses
+ example: 40101
+ metadata:
+ type: object
+ description: Additional metadata to accompany this error, such as server side stack traces or user instructions
+ nullable: true
+ required:
+ - message
+ - type
+ - code
+ examples:
+ catalogName:
+ value: prod
+ namespace:
+ value: prod.accounting
+ no_such_table_error:
+ value: |-
+ {
+ message: "The given table does not exist",
+ type: "NoSuchTableException",
+ code: 40101
+ }
+
+ no_such_iceberg_table_eror:
+ value: |-
+ {
+ message: "The given table does not exist",
Review comment:
`NoSuchIcebergTableException` is a special type of `NoSuchTableException` that indicates that the table exists, but is not an Iceberg table. For now, let's just use `NoSuchTableException` because I don't think this version of the API is going to support mixed catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754625390
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
Review comment:
I think `GET /v1/namespaces[?parent=$parent]` could work as a generic LIST namespace API endpoint, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-975911260
> Have you thought about/considered something like https://github.com/eclipse/microprofile-open-api (just an example, but there are other libs available as well), which allows you to add OpenAPI annotations directly to your Java code, so that the OpenAPI spec+docs and the code are all at one place? In combination with JAX-RS annotations, everything that defines a REST endpoint would be combined at a single place and can be easier consumed/reasoned about by developers, rather than having the Swagger definitions and the endpoint implementations across multiple files. The `swagger.json|yaml` is then generated from the annotations, which we could then serve by a separate REST endpoint (it would basically show what you see when you upload the `swagger.yaml` to editor.swagger.io).
>
> Below is a quick & short example that shows how this would look like in code:
>
> ```
> @Path("config")
> public interface ConfigurationApi {
>
> @GET
> @Produces(MediaType.APPLICATION_JSON)
> @Operation(summary = "List all catalog configuration settings")
> @APIResponses({
> @APIResponse(
> responseCode = "200",
> description =
> "All REST catalogs will be initialized by calling this route. This route\n"
> + "will return at least the minimum necessary metadata to initialize the\n"
> + "catalog. Optionally, it can also include server-side specific overrides.\n"
> + "For example, it might also include information used to initialize this catalog\n"
> + "such as the details of the Http connection pooling, etc. This route might\n"
> + "also advertise information about operations that are not implemented\n"
> + "so that the catalog can eagerly throw or go about another way of performing\n"
> + "the desired action.",
> content =
> @Content(
> mediaType = MediaType.APPLICATION_JSON,
> schema = @Schema(implementation = IcebergConfiguration.class))),
> @APIResponse(responseCode = "400", description = "Unknown Error"),
> @APIResponse(responseCode = "401", description = "Invalid credentials provided"),
> })
> IcebergConfiguration getConfig();
> }
> ```
I looked into something like this. For the moment, I chose to go without it given that I wanted to get this presentable and wasn't exactly sure how to format existing things in this way. Also, I believe we have slightly different libraries on the classpath for it.
This week is a holiday in the US (Thanksgiving) and I'm going to ry to play around with this annotation styled AP to see if I can generate the docs in this way, as I do prefer it. Thanks as always for the pointers @nastra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754719944
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
Review comment:
After looking at a longer list, I think 412 makes sense here. For commit conflicts, we can use 409: Conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754730970
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
Review comment:
Agreed. 412 for NoSuchNamespaceException? And probably 409 for NotEmpty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754729914
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
Review comment:
Might want an example with `?parent=accounting.tax`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754731947
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
Review comment:
Also, I just remembered that we don't want to have an overwrite operation because it seems idempotent. If we expect concurrent writers, then we want it to be able to retry. That means having good semantics that allow retries. Merging properties can be retried, but overwriting could easily drop properties that were added concurrently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754456290
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,763 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalogs/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
Review comment:
Yeah, I think we should drop the purge part from the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754612789
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ # TODO - Make a canonical example for NamespaceNotFound
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces/list:
Review comment:
Ok I removed it. For listing namespaces, it's currently
`GET /v1/namespaces[?namespacce=$ns]` where the query parameter `namespace` is optional. This is to allow for users to list all top level namespaces or to list namespaces contained within a given namespace.
If we prefer to have `/v1/namespaces` and `/v1/namespaces/{namespace}`, I'll have to write them up as two different routes, but it is doable. Please let me know what you think.
I don't currently like the inconsistency of having `namespace` be a query parameter here, but as a path parameter it's required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-975911260
> Have you thought about/considered something like https://github.com/eclipse/microprofile-open-api (just an example, but there are other libs available as well), which allows you to add OpenAPI annotations directly to your Java code, so that the OpenAPI spec+docs and the code are all at one place? In combination with JAX-RS annotations, everything that defines a REST endpoint would be combined at a single place and can be easier consumed/reasoned about by developers, rather than having the Swagger definitions and the endpoint implementations across multiple files. The `swagger.json|yaml` is then generated from the annotations, which we could then serve by a separate REST endpoint (it would basically show what you see when you upload the `swagger.yaml` to editor.swagger.io).
>
> Below is a quick & short example that shows how this would look like in code:
>
> ```
> @Path("config")
> public interface ConfigurationApi {
>
> @GET
> @Produces(MediaType.APPLICATION_JSON)
> @Operation(summary = "List all catalog configuration settings")
> @APIResponses({
> @APIResponse(
> responseCode = "200",
> description =
> "All REST catalogs will be initialized by calling this route. This route\n"
> + "will return at least the minimum necessary metadata to initialize the\n"
> + "catalog. Optionally, it can also include server-side specific overrides.\n"
> + "For example, it might also include information used to initialize this catalog\n"
> + "such as the details of the Http connection pooling, etc. This route might\n"
> + "also advertise information about operations that are not implemented\n"
> + "so that the catalog can eagerly throw or go about another way of performing\n"
> + "the desired action.",
> content =
> @Content(
> mediaType = MediaType.APPLICATION_JSON,
> schema = @Schema(implementation = IcebergConfiguration.class))),
> @APIResponse(responseCode = "400", description = "Unknown Error"),
> @APIResponse(responseCode = "401", description = "Invalid credentials provided"),
> })
> IcebergConfiguration getConfig();
> }
> ```
I looked into something like this. For the moment, I chose to go without it given that I wanted to get this presentable and wasn't exactly sure how to format existing things in this way. Also, I believe we have slightly different libraries already available for it, though I think those are coming form other modules.
This week is a holiday in the US (Thanksgiving) and I'm going to ry to play around with this annotation styled AP to see if I can generate the docs in this way, though I'm not sure if we'd want to keep the annotations in the Java code or not.
Thanks as always for the pointer though @nastra. Your example will help as typically I've seen this used with annotations that are more specific to Spring or some particular framework. Even for just generating the initial document, this will be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754722593
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
Review comment:
Capitalize descriptions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754732991
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
Review comment:
:smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754732560
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
Review comment:
`RemoveNamespacePropertiesRequest`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r754732158
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
Review comment:
Will be removed, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765386600
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name
+ description:
+ Rename a table from one identifier to another. It's valid to move a table
+ across namespaces, but the server implementation doesn't need to support it.
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '404':
+ description:
+ Not Found
+ - NoSuchTableException, Table to rename does not exist
+ - NoSuchNamespaceException, The target namespace of the new table identifier does not exist
+ content:
+ application/json:
+ schema:
+ oneOf:
+ - $ref: '#/components/responses/NoSuchTableResponse'
+ - $ref: '#/components/responses/NoSuchNamespaceResponse'
+ examples:
+ table_to_rename_does_not_exist:
+ value: {
+ "error": {
+ "message": "Table to rename does not exist",
+ "type": "NoSuchTableException",
+ "code": 404
+ } }
+ namespace_to_rename_to_does_not_exist:
+ value: {
+ "error": {
+ "message": "Namespace to rename to does not exist",
+ "type": "NoSuchNameSpaceException",
+ "code": 404
+ } }
+ '406':
+ description: Not Acceptable (UnsupportedOperationException)
+ '409':
+ description: Conflict (AlreadyExistsException - The new target table identifier already exists)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example: {
+ "error": {
+ "message": "Table already exists",
+ "type": "AlreadyExistsException",
+ "code": 409
+ } }
+
+components:
+
+ ##############################
+ # Application Schema Objects #
+ ##############################
+ schemas:
+ ResponseDataObject:
+ type: object
+ description: JSON data payload returned in a successful response body
+ properties:
+ data:
+ type: object
+ description: Wrapper for the response of a successful request
+ example: { "data": { "identifiers": [ "office.employees", "office.dogs", "office.cats" ] } }
+ ResponseErrorObject:
+ type: object
+ description: JSON error payload returned in a response with further details on the error
+ required:
+ - message
+ - type
+ - code
+ properties:
+ message:
+ type: string
+ description: Human-readable error message
+ type:
+ type: string
+ description: Internal type of the error, such as an exception class
Review comment:
Well, exception classes from inside of Iceberg was what I meant when I wrote that. In order for the catalog client to make use of them, they do need to be standardized.
A separate field where people can add more a more specific internal type if they wanted - that the client might not use (except generically - e.g. adding to spans or eventual metrics or something).
But "exception message" was a poor choice of words on my part. That was just meant as an example. I'll remove it. It would just be a list of well known type (at least the well-known ones the client could react to based on its value). If an implementation chose to use a non-standard value, that wouldn't be an issue but the client couldn't take an action based off of that, simply log it etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r765380221
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,657 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description: >
+ All REST catalog clients will first call this route to get possible catalog-specific
+ configuration values provided by the server, that the catalog (and its HTTP client)
+ can use to complete the `initialize` step.
+
+ This call is similar to the initial set-up calls that some catalogs already do for
+ domain-specific information, such as the Nessie catalog or the Glue catalog.
+ This is to allow for services that would like to integrate with Iceberg to do so,
+ and to be able to add their own domain-specific information into the REST catalog without
+ requiring them to write and distribute a catalog themselves.
+
+ There will be two sets of values provided -
+
+ - overrides
+ * An object containing values that the client must use.
+ For example, auth headers that the client will receive from the server
+ as temporary credentials.
+ - defaults
+ * Catalog-specific configuration that the client may use as a default value.
+ These are optional and the client is free to use its own value for these.
+
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ example: {
+ "data": {
+ "overrides": {
+ "prefix": "/raul",
+ "headers": {
+ "User-Agent": "Raul",
+ "Authorization": "Basic Ym9zY236Ym9zY28=",
+ }
+ },
+ "defaults": {
+ "clients": 5,
+ "headers": {
+ "Upgrade-Insecure-Requests": "1"
+ }
+ }
+ }
+ }
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+
+
+ /namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Set or remove properties on a namespace
+ operationId: setProperties / removeProperties
+ description:
+ Set and/or remove a collection or properties on a namespae.
+ The request body specifies a list of properties to remove and a map
+ of key value pairs to update.
+
+ Properties that are not in the request are not modified or removed by this call.
+ Server implementations are not required to support namespace properties.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdatePropertiesRequest'
+ example:
+ { "toRemove": [ "foo", "bar" ], "toUpdate": { "owner": "Raoul" } }
+ required: true
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ # This is an IcebergResponseObject<SetPropertiesResponse>, in that it uses the standard wrapper of { "data": toJson(SetPropertiesResponse) }
+ $ref: '#/components/schemas/UpdatePropertiesResponse'
+ example: {
+ "data": {
+ "updated": [ "owner" ],
+ "removed": [ "foo" ],
+ "notPresent": [ "bar" ]
+ }
+ }
+ 404:
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ { "error": { "message": "Namespace does not exist", "type": "NoSuchNamespaceException", "code": 404 } }
+ 406:
+ description: Not Acceptable (Unsupported Operation)
+ 422:
+ description: Unprocessable Entity. A property key was included in both toRemove and toUpdate.
+
+ /namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: A namespace identifier under which to list tables
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ '404':
+ description: Not Found (NoSuchNamespaceException)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 404 }'
+
+ /namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: A namespace identifier
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: A table name
+ required: true
+ schema:
+ type: string
+ example: "sales"
+
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ 200:
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ $ref: '#/components/schemas/DropTableResponse'
+ example: { "data": { "dropped": true, "purged": false } }
+ 202:
+ # TODO - Add a callback or handle to check on the state of an async purge request
+ description: Accepted - for use if purgeRequested is implemented as an asynchronous action.
+ 404:
+ $ref: '#/components/responses/NoSuchTableResponse'
+
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace.
+ responses:
+ 200:
+ description: OK - Table Exists
+ 401:
+ description: Unauthorized
+ 404:
+ description: Not Found
+
+ /tables/rename:
Review comment:
This was discussed (and I had it as that at one point).
Several people felt it was somewhat of an overloading of the API and would require people to look up the documentation of what some of the verbs do (i.e. what does PUT on table mean, without having to look it up). There was also concern that many backend frameworks can't handle a request body that might have multiple forms.
At one point I did have this implemented as a method of the original table, but that is one school of thought.
I personally don't want to utilize REST semantics for purity sake, over making a useful API for people. I don't mean that using `PUT /namespaces/{ns}/table/{tbl}` with the new identifier would cause that, but I am advocating for balancing practicality with adhering to standards.
That was the discussion that got moved to `/dev/null` by Github due to a lot of conversation so I thought I'd summarize. 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] laurentgo commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
laurentgo commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r766755796
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
Review comment:
Sorry about reopening something discussed previously. I guess that's one of the shortcomings of Github reviews which hide previous conversations?
That said, I don't necessarily see this as incompatible as a successful operation can still return a body? But what I would to be valuable is that the HTTP status code would actually give a clear indication if the operation was successful or not, similar to other operations (like when `404` is returned when the namespace or the table doesn't exist)?
I guess maybe my grief is with the iceberg API and the use of a boolean return value for `Catalog#dropTable(TableIdentifier)` to notify of an error condition whereas `Catalog#createTable(TableIdentifier, Schema)` would throw an exception? Using a return value could cause the error to go unnoticed if the caller is not checking the value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-990147673
I appreciate all of the feedback so far. Wanted to let you all know that I've been out sick so I might be a little slow in getting back to you, but I will!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r769859433
##########
File path: core/src/main/java/org/apache/iceberg/rest/http/IcebergHttpResponse.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
+ * `error` and `data`, which are themselves JSON objects.
+ *
+ * data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
+ * error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
+ *
+ * All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
+ * we'd get a JSON object back like the following:
+ *
+ * { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] }, "error", {} }
+ *
+ * If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
+ * where the `code` 40401 is a two-part identifier:
+ * - HTTP response code: 404
+ * - Two digit internal application defined error code for further detail: 01 - Namespace not found.
+ *
+ * { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
+ *
+ * We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
+ * error codes to be very helpful previously, but am open to discussion on this.
Review comment:
> Re: type values, I personally tend to think that if the field itself is part of the spec, the spec would have to define how to interpret it. I am not sure keeping it "opaque" is going to be that much useful from the spec's perspective.
I agree. Since we're using code to just be the HTTP status response code, I would say that `type` should be a well defined set we can look up.
We might also choose to allow people to use others that will get ignored by the client but are logged or something, but for now I would say let's plan for it to be a well defined set of values, but not necessarily attempt to enumerate all of them yet since we don't have about half of the basic Catalog AP in here yet. It's currently mostly just around namespaces and some of the table related ones will be handled next.
Sound good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r768116334
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,633 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: https://{host}:{port}/{basePath}
+ variables:
+ host:
+ description: The host address for the specified server
+ default: localhost
+ port:
+ description: The port used when addressing the host
+ default: "443"
+ basePath:
+ default: v1
+ - url: http://127.0.0.1:1080/v1
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalog clients will first call this route to get some configuration provided by the server.
+ This route will return any server specified default configuration values for the catalog, such as
+ configuration values used to setup the catalog for usage with Spark (e.g. vectorization-enabled).
+
+ Users should be able to override these values with client specified values.
+
+ The server might be able to request that the client use its value over a value that has been
+ configured in the client application. How and if it will do that is an open question, and thus
+ not currently specified in this documents schema.
+ responses:
+ default:
+ description: Server-Specific Configuration Values (or Overrides)
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+ /namespaces:
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ example: ?parent=accounting.tax
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found (Parent namespace does not exist)
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties, such as last_modified_time etc.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ '409':
+ $ref: '#/components/responses/NamespaceAlreadyExistsResponse'
+
+ /namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '404':
+ $ref: '#/components/responses/NoSuchNamespaceResponse'
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a namespace exists
+ operationId: namespaceExists
+ description: Check if a namespace exists.
+ responses:
+ '200':
+ description: OK - Namesapce exists
+ '401':
+ description: Unauthorized
+ '404':
+ description: Not Found
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/responses/IcebergResponseObject'
+ example: { "data": { "dropped": true } }
Review comment:
> I guess maybe my grief is with the iceberg API and the use of a boolean return value for Catalog#dropTable(TableIdentifier) to notify of an error condition whereas Catalog#createTable(TableIdentifier, Schema) would throw an exception? Using a return value could cause the error to go unnoticed if the caller is not checking the value.
That's fair. So this is sort of just a sample body. We could use a 4xx here on failure, to ensure that it's not missed. And then the REST catalog could translate the correct 4xx into the drop failing.
For things that are majorly issues / concerns with the Iceberg API in general, I'd suggest possibly opening a separate issue for that. The focus here is just to be able to get started on some sort of generalized enough HTTP API so that people who didn't want to write their own catalog could plug into this one (which has come up a few times from various people).
For this specific one, the 200 is really what indicates the drop was successful. The body is just additional metadata really (so that it could be evolved without having to change from not parsing a body to parsing a body.
That said, I agree that some sort of 4xx should be thrown if the purge was requested and didn't work. What would you suggest as far as 4xx responses go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick closed pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick closed pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757709087
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog
+ operationId: dropTable
+ description: Remove a table from the catalog
+ parameters:
+ - name: purgeRequested
+ in: query
+ required: false
+ description: Whether the user requested to purge the underlying table's data and metadata
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ description: true if the table was dropped and false if the table did not exist
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/namespaces/{namespace}/tables:
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to list tables.
+ in: path
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ required: false
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ put:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # Not using 404 as 404 is very hard to monitor.
+ # There's no way to tell by monitoring returned response codes
+ # if a client made a valid request but the table didn't exist,
+ # or if a client or server has been misconfigured or some bug exists
+ # and the client is actually calling endpoints that the server doesn't
+ # have.
+ #
+ # We can consider a different response code to use than 412.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: Load the metadata properties for a namespace
+ operationId: loadNamespaceMetadata
+ description: Return all stored metadata properties for a given namespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application:
+ schema:
+ $ref: '#/components/schemas/GetNamespaceResponse'
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ put:
+ tags:
+ - Catalog API
+ summary: Add or overwrite properties to an existing namespace
+ operationId: setNamespaceProperties
+ description:
+ Adds propertiess for a namespace. This will overwrite any existing properties,
+ and merge with the others.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RemovePropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ '417':
+ description: Namespace not found
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchNamespaceError'
+ example:
+ '{ error: { message: "Namespace does not exist", type: "NoSuchNamespaceException", code: 41701 }'
+ '409':
+ description: Namespace already exists
+ content:
+ applicaton/json:
+ schema:
+ $ref: '#/components/schemas/NamespaceAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40901 }'
+ post:
+ tags:
+ - Catalog API
+ summary: Overwrite a namespace's properties with a new set of properties
+ description: Set properties on a namespace
+ operationId: setProperties
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/SetPropertiesRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: object
+ example: '{ data: { success: true }, error: { } }'
+ # TODO - Pagination
+ /v1/namespaces:
+ parameters:
+ - name: parent
+ in: query
+ description: Optional parent namespace under which to list namespaces. When empty, list top-level namespaces.
+ required: false
+ schema:
+ type: string
+ get:
+ tags:
+ - Catalog API
+ summary: List namespaces, optionally providing a parent namespace to list underneaath
+ description:
+ List all namespaces at a certain level, optionally starting from a given parent namespace.
+ For example, if table a.b.t exists, using 'SELECT NAMESPACE IN a' this would translate into
+ `GET /v1/namespaces?parent=a` and must return Namepace.of("a","b").
+ operationId: listNamespaces
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListNamespacesResponse'
+ '401':
+ description: Unauthorized
+ /v1/namespaces/{namespace}:
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ post:
+ tags:
+ - Catalog API
+ summary: Create a namespace
+ description: Create a namespace, with an optional set of properties. The server might also add properties.
+ operationId: createNamespace
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateNamespaceRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a namespace from the catalog. Namespace must be empty.
+ operationId: dropNamespace
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ type: boolean
+ # TODO - Not empty exception, No such namespace exception
+ '401':
+ description: Unauthorized
+components:
+ schemas:
+ CommitTableRequest:
+ type: object
+ properties:
+ tableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ metadataJson:
+ type: string
+ TableIdentifier:
+ type: object
+ required:
+ - namespace
+ properties:
+ namespace:
+ type: array
+ items:
+ type: string
+ nullable: false # Note this is ignored
+ name:
+ pattern: \S
+ type: string
+ nullable: false
+ CommitTableResponse:
+ type: object
+ properties:
+ metadataLocation:
+ type: string
+ metadataJson:
+ type: string
+ RemovePropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ uniqueItems: true
+ type: array
+ items:
+ type: string
+ Catalog:
+ type: object
+ properties:
+ id:
+ type: string
+ format: uuid
+ description: Unique identifier for this catalog
+ name:
+ type: string
+ location:
+ type: string
+ description: Warehouse location for this catalog or URI of metastore or other identifying location
+ properties:
+ type: object
+ description: Additional catalog level properties
+ default: {}
+ required:
+ - name
+ - properties
+ CreateNamespaceRequest:
+ type: object
+ properties:
+ namespace:
+ type: array
+ description: individual levels of the namespace
+ items:
+ type: string
+ properties:
+ type: object
+ description: Configuration properties for the namespace
+ example: '{ owner: "Hank Bendickson" }'
+ RenameTableRequest:
+ type: object
+ properties:
+ sourceTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ destinationTableIdentifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ CreateTableRequest:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ schema:
+ $ref: '#/components/schemas/Schema'
+ partitionSpec:
+ $ref: '#/components/schemas/PartitionSpec'
+ sortOrder:
+ $ref: '#/components/schemas/SortOrder'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ metadataJson:
+ type: string
+ commit:
+ type: boolean
+ PartitionSpec:
+ type: object
+ properties:
+ unpartitioned:
+ type: boolean
+ Schema:
+ type: object
+ properties:
+ aliases:
+ type: object
+ additionalProperties:
+ type: integer
+ format: int32
+ SortOrder:
+ type: object
+ properties:
+ unsorted:
+ type: boolean
+ CreateTableResponse:
+ type: object
+ properties:
+ identifier:
+ $ref: '#/components/schemas/TableIdentifier'
+ tableLocation:
+ type: string
+ description: "Path of of the table just before the standard \
+ metadata / data folders. This is useful in cases where a different location
+ provider might be used"
+ example: s3://bucket/prod/accounting.db/monthly_sales
+ metadataLocation:
+ type: string
+ description: "Location of the most current primary metadata file for the table"
+ example: s3://bucket/prod/accounting.db/monthly_sales/metadata/00001-0d4ef14f-ef7d-43e7-9af-5f4ad40a1103.metadata.json
+ metadataJson:
+ type: string
+ description: "Stringified JSON representing the tables metadata"
+ example:
+ "TODO"
+ accessToken:
+ type: string
+ SetPropertiesRequest:
+ type: object
+ properties:
+ namespace:
+ type: string
+ properties:
+ type: object
+ ListNamespacesResponse:
+ type: object
+ properties:
+ databases:
+ type: array
+ items:
+ $ref: '#/components/schemas/Namespace'
+ Namespace:
+ type: object
+ description: Reference to one or more levels of a namespace
+ properties:
+ empty:
+ type: boolean
+ GetNamespaceResponse:
+ type: object
+ properties:
+ namespace:
+ $ref: '#/components/schemas/Namespace'
+ properties:
+ type: object
+ additionalProperties:
Review comment:
This is just a commonly used way of expressing "extra" object properties.
I'll remove it as it's not needed (I checked the generated documentation) and it seems to cause more confusion than clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r757709720
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
Review comment:
I'll see if it's possible to define those once for responses that are possible on every call. It would be cleaner for sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751522591
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
why do we need a specific path for properties? I thought it could just be a part of the namespace response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r751571034
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,761 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 0.0.1
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+- BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Invalid credentials provided
+ # This might be optional for now as it's not really supported in
+ # the normal Catalog spec, but we might want to include it for
+ # convenience.
+ /v1/catalog/{catalog}:
+ parameters:
+ - name: catalog
+ in: path
+ required: true
+ description: Name of the catalog being configured
+ schema:
+ type: string
+ minLength: 1
+ post:
+ tags:
+ - Configuration API
+ summary: Persist catalog specific configuration, which can be retrieved
+ for later use.
+ operationId: postConfig
+ description:
+ Persist some catalog specific configurations, which will be returned by \
+ calls to /v1/config in the future. This is basically all of the data \
+ that would go into the Catalog's `initialize` call.
+ # TODO - Make this into a CatalogConfiguration
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Catalog'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized / Invalid Credentials Provided
+
+ # Could also consider /v1/tables/{identifier}
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ # Consider moving this to query parameters,so it can be more easily URL encoded
+ # in case of dots or special characters
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ multipart_namespace:
+ value: "prod.accounting"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ put:
+ tags:
+ - Catalog API
+ summary: Commit an in progress create (or replace) table transaction
+ operationId: commitTable
+ description: Commit a pending create (or replace) table transaction, e.g. for doCommit.
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CommitTableResponse'
+ '401':
+ description: Unauthorized / Invalid Credentials
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergResponseObject'
+ delete:
+ tags:
+ - Catalog API
+ summary: Drop a table from the catalog, optionally purging the underlying data
+ operationId: dropTable
+ description: Remove a table from the catalog, optionally dropping the underlying data
+ parameters:
+ - name: purge
+ in: query
+ required: false
+ schema:
+ type: boolean
+ default: false
+ responses:
+ '200':
+ description: OK
+ content:
+ 'application/json':
+ schema:
+ type: boolean
+ head:
+ tags:
+ - Catalog API
+ summary: Check if a table with a given identifier exists
+ operationId: tableExists
+ description:
+ Check if a table exists within a given namespace. Returns the standard response with `true` when found. Will return a TableNotFound error if not present. Can change to returning a 200 with a body of `false` if not found, but that does add more wok on the client.
+ parameters:
+ - name: namespace
+ in: path
+ required: true
+ schema:
+ type: string
+ - name: table
+ in: path
+ required: true
+ schema:
+ type: string
+ responses:
+ '200':
+ description: OK
+ '412':
+ description: Table Not Found
+ /v1/tables:
+ get:
+ tags:
+ - Catalog API
+ summary: List all table identifiers underneath a given namespace
+ description: Return all table identifiers under this namespace
+ operationId: listTables
+ parameters:
+ - name: namespace
+ description: Namespace under which to get identifiers. Can be one or more levels.
+ in: query
+ required: false
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "prod"
+ # Note that this one would need to be normalized due to the dots. Maybe the route should change.
+ multipart_namespace:
+ value: "prod.accounting"
+ # TODO - There's a much more native way to handle pagination
+ - name: limit
+ description: number of values to return in one request
+ in: query
+ required: false
+ schema:
+ type: integer
+ example: 10
+ - name: offset
+ description: Place in the response to continue from if paginating
+ in: query
+ schema:
+ type: integer
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListTablesResponse'
+ post:
+ tags:
+ - Catalog API
+ summary: Create a table with the identifier given in the body
+ operationId: createTable
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateTableResponse'
+ /v1/tables/renameTable:
+ post:
+ tags:
+ - Catalog API
+ summary: Rename a table from its current name to a new name within the same catalog
+ description: Rename a table within the same catalog
+ operationId: renameTable
+ requestBody:
+ description: Current table identifier to rename and new table identifier to rename to
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/RenameTableRequest'
+ required: true
+ responses:
+ '200':
+ description: OK
+ '401':
+ description: Unauthorized
+ # TODO - Probably 404 will cause monitoring headaches as it's very hard to monitor if a client made
+ # a valid request, but the table didn't exist, or if a client has been misconfigured or has
+ # some other sort of bug and is calling endpoints that don't exist.
+ # Need to settle on a non-404 error code.
+ '412':
+ description: Table to rename from does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
+ example:
+ '{ error: { message: "Table does not exist", type: "NoSuchTableException", code: 41202 }'
+ '409':
+ description: The new table identifier, the to table rename to, already exists.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TableAlreadyExistsError'
+ example:
+ '{ error: { message: "Namespace already exists", type: "AlreadyExistsException", code: 40902 }'
+ /v1/namespaces/{namespace}/properties:
Review comment:
Oh actually, I see. I have duplicated this functionality here in `GET /v1/namespaces/{namespace}` which is the same as this one. https://github.com/apache/iceberg/pull/3561/files#diff-111c5d0e1ee3097e79ed73a1546ec698a1ced352827c109c41ec21cf68db120eR441-R452
So the only thing we'd need to be able to support would be `setProperties` and `removeProperties`.
I will remove one or the other once we sort that out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#discussion_r760580649
##########
File path: rest_docs/rest-catalog-open-api-v0.1.yaml
##########
@@ -0,0 +1,727 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+openapi: 3.0.3
+info:
+ title: Apache Iceberg REST Catalog API
+ license:
+ name: Apache 2.0
+ url: https://www.apache.org/licenses/LICENSE-2.0.html
+ version: 1.0.0
+ description:
+ Defines the specification for the first version of the REST Catalog API. Implementations should support both Iceberg table specs v1 and v2, with priority given to v2.
+servers:
+ - url: http://127.0.0.1:1080
+ description: URL Used for Mock-Server Unit Tests
+# All routes are currently configured using an Authorization header.
+security:
+ - BearerAuth: []
+paths:
+ /v1/config:
+ get:
+ tags:
+ - Configuration API
+ summary: List all catalog configuration settings
+ operationId: getConfig
+ description:
+ All REST catalogs will be initialized by calling this route. This route
+ will return at least the minimum necessary metadata to initialize the
+ catalog. Optionally, it can also include server-side specific overrides.
+ For example, it might also include information used to initialize this catalog
+ such as the details of the Http connection pooling, etc. This route might
+ also advertise information about operations that are not implemented
+ so that the catalog can eagerly throw or go about another way of performing
+ the desired action.
+ responses:
+ default:
+ description: Server-Specific Configuration Overrides
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/IcebergConfiguration'
+ "400":
+ description: Unknown Error
+ "401":
+ description: Unauthorized
+
+ /v1/namespaces/{namespace}/tables/{table}:
+ parameters:
+ - name: namespace
+ in: path
+ description: Namespace the table is in
+ required: true
+ schema:
+ type: string
+ examples:
+ singlepart_namespace:
+ value: "accounting"
+ multipart_namespace:
+ value: "accounting.tax"
+ - name: table
+ in: path
+ description: Name of the table to load
+ required: true
+ schema:
+ type: string
+ example: "sales"
+ get:
+ tags:
+ - Catalog API
+ summary: Load a given table from a given namespace
+ operationId: loadTable
+ responses:
+ '200':
+ description: OK
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/GetTableResponse'
+ '401':
+ description: Unauthorized
+ # Using 412, `Precondition Failed`, instead of 404, as 404 makes monitoring response codes from ELBs
+ # very difficult - Hard to tell if clients or servers are misconfigured and calling non-existent routes
+ # or missing routes versus expected error cases such as NoSuchTableException (expected meaning that
+ # a person who is on call shouldn't be paged for this but 404 they might need to be).
+ '412':
+ description: NoSuchTableException
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/NoSuchTableError'
Review comment:
I think that this is the error schema, but that `NoSuchTableException` would be thrown. Right, @kbendick?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3561: [CORE] Specification for an HTTP REST catalog
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3561:
URL: https://github.com/apache/iceberg/pull/3561#issuecomment-984286915
First, thank you for the extensive review @rymurr! I truly do appreciate it.
> A few general comments from me:
>
> 1. 412 feels weird. the fact that some people may have load balancers in front and some load balancers standard config aren't clear about what 404 means isn't a great reason for a non-standard overload of a return code.
For reference, the primary load balancers I've worked with when encountering the want / desire to do this have been AWS provided. That's not meant as a dig on AWS ELBs or ALBs, but just to say that it's not a niche set of load balancers we're talking about. Amazon only advertise general 4xx and 5xx counts, but it's pretty common to have metrics from the load balancer or fronting servers (nginx, proxy layers) on specific codes out of the box and it's pretty common for people to parse specific codes from ELB logs via means that are typically outside of the normal observability stack (i.e. in-app / in-client metrics or reported events, if any).
That's not necessarily reason to not use 404, but I did want to point out that it likely affects most all of us. It's by no means a legacy issue or only a concern for a small set of tools. It could just be a misconfigured server as well. Or misconfigured DNS. A number of things can cause application metrics to be missing (including metrics from any kind of mesh network - though you're probably not in a good spot regardless then).
My other reason to avoid 404 is that it doesn't provide information about what was not present - in some cases, a namespace or a specific table might not be present from the same call and 404 for both requires additional tooling to monitor that, especially client side. Having a dedicated code for each is helpful there too. Using even specific codes in one response (409, 412, etc), can indicate more rich information and so I'm partial to that and having it documented vs a large number of 404s.
But there does seem to be pushback to this idea, so I'm willing to convert to 404 if enough people feel strongly about it. I'm going to handle that later as there's more feedback to cover, but will revert if people feel strongly.
> 2. Thinking about the API from a high level perspective (as opposed to a mapping of existing java interfaces) it seems we have two entities to consider: Catalog and Table. Catalog operations are mostly related to high level concerns: namespaces, listings. properties, broad config etc and typically don't require modification of iceberg metadata (therefore don't require locking). Whereas tables require locking, modification of the filesystem/object store etc.
I don't necessarily agree with the idea that Table requires locking and Catalog doesn't, only because I don' think this new catalog should necessarily prescribe how things are done either (just as the API doesn't currently fully prescribe how a catalog can achieve certain things). Though I do see where you're coming from and it's `Table`s that cause `Transaction`. So I don't completely disagree by any means.
Also, in some scenarios locking might be needed on the catalog, such as highly concurrent multi-tenant environments like Kyuubi or large Trino deploymens etc.
I've avoided `Transaction` for now because of it in this PR. As well as table creation. Those can be tackled separately.
> With this separation in mind (and the fact that we haven't specified in the API how to guarantee atomicity) I thought keeping the catalog api (eg `/v1/catalog`) to only deal w/ namespaces, properties etc may be a better separation. We could move the CRUD operations for tables to `/v1/table` paths and overload less the meaning of some of the operations here.
>
> Eg:
>
> 1. Post/Put/Delete/Get/Head on `v1/catalog/{namespace}` can refer to create, update properties, delete, get properties and check namespace existence
> 2. Post/Put/Delete/Get/Head on `v1/table/{namespace}/{table}` can refer to create, update table, delete, get table and check table existence
> Then 2. only has to deal w the atmoic CRUD operations and we override much less.
I'm not opposed to adding more separation on the routes, but admittedly I've moved a few of them back and forth a few times so I'll hold off momentarily and focus on cleaning up other parts (especially types) and providing more examples as well as working on the annotations for my own use in case we want to go that route. I'm actually finding it slightly more effective to just work on this document vs using the annotations, but the JSON representation of some items isn't out of the box without an additional library.
Ryan covered some reasons to have additional string constants in the path, which I agree with.
We'd need more routes than what you proposed still, such as removeProperties and setProperties (on a namespace or on a table, but two separate issues). And renameTable.
For now this PR mostly aims to cover `catalog` related issues and non-transaction based concerns on tables (so simple gets, list, and presently the main drop interface - RD from CRUD on tables).
Sorry for the long response, but thank you for your thorough review. I wanted to give people enough time to respond before working on it further given it became somewhat of a quick back and forth and then I was on vacation.
Thanks again @rymurr!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org