You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "jbonofre (via GitHub)" <gi...@apache.org> on 2026/10/30 07:47:17 UTC
[PR] Add the Directories REST API specification and documentation [polaris]
jbonofre opened a new pull request, #5927:
URL: https://github.com/apache/polaris/pull/5927
This is the first of three PRs split from #4613, as discussed during the community meeting, to make the review easier:
1. **This PR**: OpenAPI specification and documentation
2. PR2 (to be linked): Directory entity, REST endpoints and `DirectoryScanService` SPI
3. PR3 (to be linked): simple directory scan service
Directories allow users to point Polaris at a location/prefix on an object store (S3, GCS, Azure) containing objects (including unstructured data like images, videos, documents) and track the objects in an Iceberg table with object-level metadata (location, size, content type, checksum, last modified, etc.).
### Changes
- `spec/polaris-catalog-apis/directories-api.yaml`: new Directory API under `/polaris/v1/{prefix}/namespaces/{namespace}/directories` (list, create, load, drop and scan), referenced from `polaris-catalog-service.yaml`.
- `StorageAccessConfig` moves from `generic-tables-api.yaml` to a new `common-schemas.yaml`, so generic tables and directories share the same schema. The generic tables API itself is unchanged.
- `spec/generated/bundled-polaris-catalog-service.yaml` is regenerated with `spec/bundle.sh`.
- `site/content/in-dev/unreleased/directories.md`: user documentation for directories.
No server code is generated from the new API in this PR: `DirectoryApi` is added to the generated APIs in the next PR, together with its implementation.
### Notes for reviewers
- The documentation describes the whole feature, including the `ENABLE_DIRECTORIES` and `ENABLE_DIRECTORY_SCAN` feature flags, which land in the next two PRs.
- The review comments already addressed on #4613 are included.
## Checklist
- [X] ๐ก๏ธ Don't disclose security issues! (contact security@apache.org)
- [X] ๐ Clearly explained why the changes are needed, or linked related issues: Related to #4613
- [X] ๐งช Added/updated tests with good coverage, or manually tested (and explained how)
- [X] ๐ก Added comments for complex logic
- [X] ๐งพ Updated `CHANGELOG.md` (if needed)
- [X] ๐ Updated documentation in `site/content/in-dev/unreleased` (if needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4460487100
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
No worries, this is more of an opinion on my part (non-blocking).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4528514226
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Load a directory from the catalog under the given namespace.
+
+ The response contains the directory configuration and, when the directory inventory table exists,
+ its identifier.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Update the mutable configuration of a directory (`scanner-config` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new scanner configuration applies from the next scan.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ type: string
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ schemas:
+ DirectoryPageToken:
+ description: >
+ An opaque token used to paginate the list of directories. The server returns it as
+ `next-page-token` when more results may be available, and the client passes it unchanged as the
+ `pageToken` query parameter to fetch the next page. The client does not need to send a
+ `pageToken` to start paginating: any response may be paginated, whether or not the request
+ carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no
+ more results.
+ type: [ string, "null" ]
+
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ DirectoryScannerConfig:
+ type: object
+ description: >
+ Configuration used by the external scanning services to populate the directory inventory table.
+ Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ UpdateDirectoryRequest:
+ type: object
+ description: >
+ Fields to update on a directory. Omitted fields are left unchanged.
+ properties:
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ Directory:
+ type: object
+ description: >
+ Directory information.
+
+ - `name` (REQUIRED): name of the directory
+
+ - `base-location` (REQUIRED): root location of the directory on the object store
+
+ - `scanner-config` (OPTIONAL): configuration for the scanning services, such as the include and exclude
+ patterns for object filtering
+
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+
+ - `inventory-table` (OPTIONAL): identifier of the directory inventory table, assigned by Polaris. It is
+ absent if the inventory table does not exist, for example if it was dropped through the Iceberg REST API.
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ inventory-table:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/TableIdentifier'
+
+ LoadDirectoryResponse:
+ description: Result used when a directory is successfully loaded.
+ type: object
+ required:
+ - directory
+ properties:
+ directory:
+ $ref: '#/components/schemas/Directory'
+
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/TableIdentifier'
+
+ responses:
+
+ LoadDirectoryResponse:
+ description: Directory result if successfully loaded a directory.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/LoadDirectoryResponse'
+
+ CreateDirectoryResponse:
+ description: Directory result if successfully created a directory.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/LoadDirectoryResponse'
+
+ ListDirectoriesResponse:
Review Comment:
line 428 defines a ListDirectoriesResponse, which refers to line 401 with the same name. Do we need two schema for the same thing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4442608261
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
I think it's semantically different.
I would prefer an opinionated endpoint specific to directories, it's clearer imho.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4469637291
##########
site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md:
##########
@@ -300,6 +300,15 @@ Flag to enable or disable the API to reset principal credentials. Defaults to en
---
+##### `polaris.features."ENABLE_DIRECTORIES"`
+
+If true, the directories endpoints are enabled and advertised to clients in the catalog configuration response. If false, the endpoints return 501 Not Implemented. This is a beta feature: the API is under active development and may change in a backward-incompatible way. It is disabled by default; enable it with caution and report any issues encountered.
Review Comment:
Yes. Thanks! If it weren't in the docs, it would be ok, but with doc exposure I think it's rather confusing to end users. +1 to add it together with impl. code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4478635286
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
Review Comment:
Thanks! The header is removed for now, as credential vending for the `base-location` isn't supported yet (see @flyrain comment on the header description). When we add it back, I will consider a query parameter or a request field as you suggest.
I will work on it once this PR is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4469721270
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
Review Comment:
optional: Header matches the existing IRC pattern, so it is fine.
However, in general, this parameter feels like a first class parameter in the API. It is not a cross-cutting concern, but affects what goes into the response payload for certain specific endpoints. It might be worth considering putting it on the request payload or query parameter.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
Review Comment:
optional: Header matches the existing IRC pattern, so it is fine.
However, in general, this parameter feels like a first class parameter in the API. It is not a cross-cutting concern, but affects what goes into the response payload for certain specific endpoints.
It might be worth considering putting it on the request payload or query parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4478646616
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
Review Comment:
`pageToken` now has an inline `type: string` schema.
I kept `DirectoryPageToken` for `next-page-token` in `ListDirectoriesResponse`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4428584768
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
Do we need a new endpoint for the directory? It is a sub type of generic table. I think we could follow the way we introduce Delta/Hudi/Paimon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4442709371
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,283 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, which objects to include, and how often to re-scan. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is managed by Polaris. Dropping it, or writing to it through the Iceberg REST API, breaks the directory:
+the next scan fails or the inventory no longer reflects the objects of the directory. Use the directories API to scan or drop a directory.
+{{< /alert >}}
+
+Polaris provides a simple synchronous scan endpoint but does not schedule scans. External services (e.g. directory table scanning service) can also read the directory configuration through the REST API,
+walk the object store, and write the results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base_location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+| `scan-schedule` | `object` | No | Object representing a scan schedule (trigger, cron, ...). |
+
+### Storage access
+
+Directories use Polaris `StorageAccessConfig` to access the object store.
+
+The `base_location` is validated like a table location when the directory is created and again before each scan:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base_location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base_location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ },
+ "scan-schedule": {
+ "cron": "0 * * * *"
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including filter and scan schedule.
+
+### Drop a directory
+
+**DELETE** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+### Scan a directory
+
+**POST** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan`
+
+Lists the objects under the directory `base_location`, applies the configured [filter](#filter) and
+replaces the content of the directory table with the result in a single commit. The response contains
+the number of objects recorded in the table:
+
+```json
+{ "file-count": 42 }
+```
+
+Polaris lists the `base_location` with read-only credentials scoped to that location, and requires the
+`UPDATE_TABLE` privilege on the directory. The scan is synchronous and does not run on the
+`scan-schedule`: it is up to the caller (or an [external service](#external-services-scanning-services))
+to trigger it periodically. If the scan fails, the directory table remains unchanged.
+
+The scan service is disabled by default: it is enabled with the `ENABLE_DIRECTORY_SCAN` feature flag (see
Review Comment:
Good point. I added an `ENABLE_DIRECTORIES` feature flag (beta, default `false`) that gates the whole Directories API, not just scan, and dropped the `ENABLE_DIRECTORY_SCAN` mention. When it is off, all the endpoints return `501 Not Implemented`, like the OpenLineage flag, and they aren't advertised in the catalog config. This is documented in `directories.md` and in the features reference. The spec's `5XX` response already covers 501. The enforcement itself will land with the implementation PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4460584354
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,478 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/page-token'
+ - $ref: '#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter`, `scan-schedule` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
Review Comment:
I think it'd be preferable to specify how page tokens are used in the Polaris YAML.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "orca-review-bot (via GitHub) via issues" <is...@polaris.apache.org>.
orca-review-bot commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4529511030
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,433 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
+
+ Polaris also creates the directory inventory table, empty, in the same namespace, and returns its
+ identifier in `inventory-table`. Creating a directory doesn't scan the `base-location`: the inventory
+ table is populated by the external scanning services.
+ operationId: createDirectory
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Load a directory from the catalog under the given namespace.
+
+ The response contains the directory configuration and, when the directory inventory table exists,
+ its identifier.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Update the mutable configuration of a directory (`scanner-config` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new scanner configuration applies from the next scan.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Remove a directory under the given namespace from the catalog, together with its inventory table.
+ The inventory table is dropped without purging its data and metadata files. The objects under the
+ directory `base-location` are never deleted.
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ type: string
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ schemas:
+ DirectoryPageToken:
+ description: >
+ An opaque token used to paginate the list of directories. The server returns it as
+ `next-page-token` when more results may be available, and the client passes it unchanged as the
+ `pageToken` query parameter to fetch the next page. The client does not need to send a
+ `pageToken` to start paginating: any response may be paginated, whether or not the request
+ carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no
+ more results.
+ type: [ string, "null" ]
+
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ DirectoryScannerConfig:
+ type: object
+ description: >
+ Configuration used by the external scanning services to populate the directory inventory table.
+ Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris. It is validated like a table location, so it must be within the catalog's allowed locations and, unless the catalog allows unstructured table locations, within the namespace location.
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ UpdateDirectoryRequest:
+ type: object
+ description: >
+ Fields to update on a directory. Omitted fields are left unchanged.
Review Comment:
"Omitted fields are left unchanged" means there's no way to remove a `scanner-config` or a filter once it's set. You'd have to drop and recreate the directory, which also drops the inventory table. Should an explicit `null` or `{}` clear the field? Can we say which one in the spec?
<sub>๐ค AI-generated review comment โ agents make mistakes, so please verify before acting on it.</sub>
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,433 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
+
+ Polaris also creates the directory inventory table, empty, in the same namespace, and returns its
+ identifier in `inventory-table`. Creating a directory doesn't scan the `base-location`: the inventory
+ table is populated by the external scanning services.
+ operationId: createDirectory
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Load a directory from the catalog under the given namespace.
+
+ The response contains the directory configuration and, when the directory inventory table exists,
+ its identifier.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Update the mutable configuration of a directory (`scanner-config` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new scanner configuration applies from the next scan.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Remove a directory under the given namespace from the catalog, together with its inventory table.
+ The inventory table is dropped without purging its data and metadata files. The objects under the
+ directory `base-location` are never deleted.
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ type: string
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ schemas:
+ DirectoryPageToken:
Review Comment:
Generic-tables and policies reuse the Iceberg `page-token`/`PageToken`, and this adds a second pagination contract to the same service. The main difference is that you don't need to send an empty token to start paging. If that's the reason, can we say so in the PR description? Otherwise, can we reuse the Iceberg parameter so the Polaris list APIs stay consistent?
<sub>๐ค AI-generated review comment โ agents make mistakes, so please verify before acting on it.</sub>
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,280 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+The directory table is a snapshot of the objects as of the last scan, not a live view of the object store. Objects
+added, changed, or removed after the last scan appear in the directory table only after the next scan, so the
+directory table can lag behind the object store.
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `scanner-config` | `object` | No | Configuration for the scanning services, such as the filter that controls which objects are added to the directory table during a scan. See [Scanner configuration](#scanner-configuration). |
Review Comment:
The spec also accepts `properties` on create, and the update section mentions it, but this table doesn't list it. Can we add a row?
<sub>๐ค AI-generated review comment โ agents make mistakes, so please verify before acting on it.</sub>
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,433 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
+
+ Polaris also creates the directory inventory table, empty, in the same namespace, and returns its
+ identifier in `inventory-table`. Creating a directory doesn't scan the `base-location`: the inventory
+ table is populated by the external scanning services.
+ operationId: createDirectory
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Load a directory from the catalog under the given namespace.
+
+ The response contains the directory configuration and, when the directory inventory table exists,
+ its identifier.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
Review Comment:
A missing directory is reported as `NoSuchTableError`. Clients that branch on the error `type` then can't tell "directory not found" from "table not found". The 409 at line 117 has the same problem. Is that the type the server will return in #5928, or should directories get their own error type while the API is still beta?
<sub>๐ค AI-generated review comment โ agents make mistakes, so please verify before acting on it.</sub>
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,280 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+The directory table is a snapshot of the objects as of the last scan, not a live view of the object store. Objects
+added, changed, or removed after the last scan appear in the directory table only after the next scan, so the
+directory table can lag behind the object store.
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `scanner-config` | `object` | No | Configuration for the scanning services, such as the filter that controls which objects are added to the directory table during a scan. See [Scanner configuration](#scanner-configuration). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Scanner configuration
+
+The `scanner-config` object holds the configuration used by the
+[external scanning services](#external-services-scanning-services) to populate the directory table. Polaris stores
+it and returns it when the directory is loaded, but does not act on it.
+
+It contains a [`filter`](#filter) that controls which objects are added to the directory table during a scan.
+
+#### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "scanner-config": {
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "scanner-config": {
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+The [directory table](#directory-table) is always created, empty, with the directory, and its identifier is
+returned in the `inventory-table` field. Creating a directory doesn't scan the `base-location`: the directory
+table is populated by the [external scanning services](#external-services-scanning-services).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its scanner configuration, and the
+identifier of its [directory table](#directory-table) in the `inventory-table` field. The `inventory-table` field
+is absent if the directory table does not exist.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `scanner-config` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new scanner configuration applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace. The directory table is dropped
+without purging its data and metadata files. The objects under the directory `base-location` are never deleted.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table identifier is also returned in the `inventory-table`
+field when the directory is loaded. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). It is set per object because objects under the same location can have checksums computed with different algorithms (for example S3 objects uploaded with different checksum settings), so a scanning service can record the checksum reported by the object store without reading the object. |
+| 5 | `checksum` | `string` | No | The object checksum value computed with the algorithm specified in `checksum_algorithm`. |
+| 6 | `last_modified` | `timestamptz` | Yes | The last modification timestamp of the object as reported by the object store. |
+| 7 | `metadata` | `map<string, string>` | No | Additional labels and tags from the object store (for example S3 user-defined metadata, storage class, or content encoding). |
+
+This schema is part of the Directories API specification. Its fields are not renamed or removed, and new fields
+are only added as optional fields, so existing clients can keep reading the directory table.
+
+Clients (query engines, analytics tools, etc.) interact with the directory table as a regular Iceberg
+table. For instance, you can query it with Spark SQL:
+
+```sql
+SELECT file_uri, size, last_modified
+FROM my_catalog.my_namespace.`product-images__inventory`
+WHERE content_type = 'image/jpeg'
+ AND size > 1048576
+ORDER BY last_modified DESC;
+```
+
+### Credential vending
+
+The directory table lists the objects, it does not give access to them. Reading the directory table returns the
+object URIs and metadata, but a user or engine needs its own access to the object store to read the objects
+themselves.
+
+More precisely:
+
+- When an engine loads the directory table with [credential vending]({{% ref "vended-credentials" %}}), Polaris
+ vends credentials scoped to the directory table location (its data and metadata files), as for any Iceberg
+ table. These credentials are not scoped to the directory `base-location`.
+- Privileges on the directory or on the directory table (for example `TABLE_READ_DATA`) don't grant access to the
Review Comment:
This mentions "privileges on the directory", but the page never says which privileges govern list/create/load/update/drop. Can we name them here, or say they'll be defined in #5928, so admins know what to grant?
<sub>๐ค AI-generated review comment โ agents make mistakes, so please verify before acting on it.</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4465089892
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
I'm still in favor of different endpoints because:
- `filter` and `scan-schedule` are typed. As a generic table, they would become untyped string properties.
- There's a `scan` action, which generic tables don't have.
- What users query is an Iceberg table (`__inventory`), and generic tables are defined as non-Iceberg.
- `base-location` points at existing external data the catalog doesn't own.
If the implementation PR (phase 2) can reuse the generic-table code for location validation, overlap checks and credential vending, keeping the API separate doesn't mean rebuilding everything.
Should a directory be allowed over a prefix that overlaps an existing table's location? Generic table rules would reject that, and it's plausible use case.
So, I'm still in favor of Directory endpoint, not generic table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4469614522
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
I think a separate Directories endpoint makes sense.
* It defines a specific and focused contract.
* It defines functionality beyond the scope of Generic Tables.
* It evolves independently of the Generic Tables API.
* It allows downstream projects to elect implementing Directories with or without the broader Generic Tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4478598609
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryPageToken:
+ description: >
+ An opaque token used to paginate the list of directories. The server returns it as
+ `next-page-token` when more results may be available, and the client passes it unchanged as the
+ `pageToken` query parameter to fetch the next page. The client does not need to send a
+ `pageToken` to start paginating: any response may be paginated, whether or not the request
+ carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no
+ more results.
+ type: [ string, "null" ]
+
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ UpdateDirectoryRequest:
+ type: object
+ description: >
+ Fields to update on a directory. Omitted fields are left unchanged.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ Directory:
+ type: object
+ description: >
+ Directory information.
+
+ - `name` (REQUIRED): name of the directory
+
+ - `base-location` (REQUIRED): root location of the directory on the object store
+
+ - `filter` (OPTIONAL): include and exclude patterns for object filtering
+
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ LoadDirectoryResponse:
+ description: Result used when a directory is successfully loaded.
+ type: object
+ required:
+ - directory
+ properties:
+ directory:
+ $ref: '#/components/schemas/Directory'
+ storage-access-configs:
+ type: array
+ description: |
+ Storage configurations required to access the directory, encompassing both credentials and other necessary
+ settings for storage access, such as the client region for S3.
+ items:
+ $ref: './common-schemas.yaml#/components/schemas/StorageAccessConfig'
+
+ ScanDirectoryResponse:
+ description: Result of a directory scan.
+ type: object
+ required:
+ - file-count
+ properties:
+ file-count:
+ type: integer
+ format: int64
+ description: Number of objects recorded in the directory inventory table after the scan.
Review Comment:
Agreed. I removed the scan endpoint for now.
If we bring it back, an async model will be part of that discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4478587353
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
Review Comment:
That's fair and good point.
I removed the scan endpoint for now.
Polaris now stores the directory configuration and manages the inventory table lifecycle (created and dropped with the directory), while scanning and scheduling are done by external services, as described in `directories.md`. This also avoids having two writers on the inventory with different semantics (a full replace vs incremental updates).
Once we agree on the execution and ownership model, we can discuss in a follow-up whether a scan operation is needed and how callers observe its completion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4500206656
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
Review Comment:
Good point. I added a paragraph to the overview: the directory table is a snapshot as of the last scan, not a live view, and can log behind the object store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "ayushtkn (via GitHub) via issues" <is...@polaris.apache.org>.
ayushtkn commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4430844064
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,283 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, which objects to include, and how often to re-scan. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is managed by Polaris. Dropping it, or writing to it through the Iceberg REST API, breaks the directory:
+the next scan fails or the inventory no longer reflects the objects of the directory. Use the directories API to scan or drop a directory.
+{{< /alert >}}
+
+Polaris provides a simple synchronous scan endpoint but does not schedule scans. External services (e.g. directory table scanning service) can also read the directory configuration through the REST API,
+walk the object store, and write the results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base_location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+| `scan-schedule` | `object` | No | Object representing a scan schedule (trigger, cron, ...). |
+
+### Storage access
+
+Directories use Polaris `StorageAccessConfig` to access the object store.
+
+The `base_location` is validated like a table location when the directory is created and again before each scan:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base_location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base_location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ },
+ "scan-schedule": {
+ "cron": "0 * * * *"
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/v1/{prefix}/namespaces/{namespace}/directories`
Review Comment:
Endpoint paths are missing the `/polaris` prefix. The spec defines `/polaris/v1/{prefix}/namespaces/{namespace}/directories`. `generic-table.md` and `policy.md` both write the full `/polaris/v1/{prefix}/....`
Same issue at 99 (create example) and 237, 239 (mermaid)
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,283 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, which objects to include, and how often to re-scan. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is managed by Polaris. Dropping it, or writing to it through the Iceberg REST API, breaks the directory:
+the next scan fails or the inventory no longer reflects the objects of the directory. Use the directories API to scan or drop a directory.
+{{< /alert >}}
+
+Polaris provides a simple synchronous scan endpoint but does not schedule scans. External services (e.g. directory table scanning service) can also read the directory configuration through the REST API,
+walk the object store, and write the results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base_location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+| `scan-schedule` | `object` | No | Object representing a scan schedule (trigger, cron, ...). |
+
+### Storage access
+
+Directories use Polaris `StorageAccessConfig` to access the object store.
+
+The `base_location` is validated like a table location when the directory is created and again before each scan:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base_location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base_location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ },
+ "scan-schedule": {
+ "cron": "0 * * * *"
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including filter and scan schedule.
+
+### Drop a directory
+
+**DELETE** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+### Scan a directory
+
+**POST** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan`
+
+Lists the objects under the directory `base_location`, applies the configured [filter](#filter) and
+replaces the content of the directory table with the result in a single commit. The response contains
+the number of objects recorded in the table:
+
+```json
+{ "file-count": 42 }
+```
+
+Polaris lists the `base_location` with read-only credentials scoped to that location, and requires the
+`UPDATE_TABLE` privilege on the directory. The scan is synchronous and does not run on the
Review Comment:
`UPDATE_TABLE` is not a `Polaris` privilege โ it's a `PolarisAuthorizableOperation`. `PolarisPrivilege` has no such value; the grantable table privileges are `TABLE_WRITE_DATA`, `TABLE_ADD_SNAPSHOT`, `TABLE_WRITE_PROPERTIES`, etc. A reader can't grant what this says to grant, or am I missing something here?
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,283 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, which objects to include, and how often to re-scan. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is managed by Polaris. Dropping it, or writing to it through the Iceberg REST API, breaks the directory:
+the next scan fails or the inventory no longer reflects the objects of the directory. Use the directories API to scan or drop a directory.
+{{< /alert >}}
+
+Polaris provides a simple synchronous scan endpoint but does not schedule scans. External services (e.g. directory table scanning service) can also read the directory configuration through the REST API,
+walk the object store, and write the results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base_location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+| `scan-schedule` | `object` | No | Object representing a scan schedule (trigger, cron, ...). |
+
+### Storage access
+
+Directories use Polaris `StorageAccessConfig` to access the object store.
+
+The `base_location` is validated like a table location when the directory is created and again before each scan:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base_location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base_location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ },
+ "scan-schedule": {
+ "cron": "0 * * * *"
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including filter and scan schedule.
+
+### Drop a directory
+
+**DELETE** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+### Scan a directory
+
+**POST** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan`
+
+Lists the objects under the directory `base_location`, applies the configured [filter](#filter) and
+replaces the content of the directory table with the result in a single commit. The response contains
+the number of objects recorded in the table:
+
+```json
+{ "file-count": 42 }
+```
+
+Polaris lists the `base_location` with read-only credentials scoped to that location, and requires the
+`UPDATE_TABLE` privilege on the directory. The scan is synchronous and does not run on the
+`scan-schedule`: it is up to the caller (or an [external service](#external-services-scanning-services))
+to trigger it periodically. If the scan fails, the directory table remains unchanged.
+
+The scan service is disabled by default: it is enabled with the `ENABLE_DIRECTORY_SCAN` feature flag (see
+[Configuration]({{% ref "configuration/configuration-reference" %}})). When it is disabled, the endpoint is rejected and not
+advertised in the catalog configuration. The scan populates `file_uri`, `content_type` (guessed from the
+file name), `size` and `last_modified` (as reported by the object store); the other columns are left empty.
+It requires an object store whose `FileIO` supports listing a prefix.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). |
+| 5 | `checksum` | `string` | No | The object checksum value computed with the algorithm specified in `checksum_algorithm`. |
+| 6 | `last_modified` | `timestamptz` | No | The last modification timestamp of the object as reported by the object store. |
+| 7 | `metadata` | `map<string, string>` | No | Additional labels and tags from the object store (for example S3 user-defined metadata, storage class, or content encoding). |
+
+Clients (query engines, analytics tools, etc.) interact with the directory table as a regular Iceberg
+table. For instance, you can query it with Spark SQL:
+
+```sql
+SELECT file_uri, size, last_modified
+FROM my_catalog.my_namespace.product_images
Review Comment:
Example queries the wrong table. Line 46 and 169 say the `inventory` table for `product-images` is `product-images__inventory`, but this reads `my_catalog.my_namespace.product_images`.
Also, a hyphenated name needs `backtick-quoting` in `Spark` SQL
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: Conflict - The directory already exists under the given namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base_location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base_location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ ScanSchedule:
+ type: object
+ description: >
+ Describes when an external scanning service should re-scan the directory.
+ properties:
+ cron:
+ type: string
+ description: Cron expression describing when scans should run (e.g. `0 * * * *` for the top of every hour).
+
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base_location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base_location:
Review Comment:
I think `base_location` should be `base-location`. Every other field in the Polaris specs is kebab-case โ `base-location` in `generic-tables-api.yaml`, plus `metadata-location`, `table-name`, `table-uuid`, `notification-type`, `policy-type`, `current-policy-version`, `entity-version`, `next-page-token`.
This file is also internally inconsistent: `scan-schedule`, `file-count`, `next-page-token` and `storage-access-configs` are kebab-case.
It's in required, so it's frozen once clients are generated, and the sibling API already named the same concept `base-location`. Also at 312, 316 (Directory) and in prose at 190, 253, 303, plus directories.md:63
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: Conflict - The directory already exists under the given namespace
Review Comment:
The derived __inventory name isn't in the error contract. This `409` only covers "the directory already exists". What happens when `<name>__inventory` is already taken by a regular table โ which a client can create through the `Iceberg` API at any time, and which the docs warn breaks the directory?
Is __inventory a reserved suffix that `createTable` rejects? Worth settling here since the entity lands in PR 2.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: Conflict - The directory already exists under the given namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base_location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base_location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ ScanSchedule:
+ type: object
+ description: >
+ Describes when an external scanning service should re-scan the directory.
+ properties:
+ cron:
+ type: string
+ description: Cron expression describing when scans should run (e.g. `0 * * * *` for the top of every hour).
+
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base_location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base_location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ scan-schedule:
+ $ref: '#/components/schemas/ScanSchedule'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ Directory:
+ type: object
+ description: >
+ Directory information.
+
+ - `name` (REQUIRED): name of the directory
+
+ - `base_location` (REQUIRED): root location of the directory on the object store
+
+ - `filter` (OPTIONAL): include and exclude patterns for object filtering
+
+ - `scan-schedule` (OPTIONAL): schedule describing when an external service should re-scan the directory
+
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+ required:
+ - name
+ - base_location
+ properties:
+ name:
+ type: string
+ base_location:
+ type: string
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ scan-schedule:
+ $ref: '#/components/schemas/ScanSchedule'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ LoadDirectoryResponse:
+ description: Result used when a directory is successfully loaded.
+ type: object
+ required:
+ - directory
+ properties:
+ directory:
+ $ref: '#/components/schemas/Directory'
+ storage-access-configs:
+ type: array
+ description: |
+ Storage configurations required to access the directory, encompassing both credentials and other necessary
+ settings for storage access, such as the client region for S3.
+ items:
+ $ref: './common-schemas.yaml#/components/schemas/StorageAccessConfig'
+
+ ScanDirectoryResponse:
+ description: Result of a directory scan.
+ type: object
+ required:
+ - file-count
+ properties:
+ file-count:
+ type: integer
+ format: int64
+ description: Number of objects recorded in the directory inventory table after the scan.
+
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
+ identifiers:
Review Comment:
`identifiers` is a list of `TableIdentifier` โ does `listDirectories` return `product-images` or `product-images__inventory`? Maybe worth one line somewhere.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: Conflict - The directory already exists under the given namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base_location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base_location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
Review Comment:
Just a thought-
Regex dialect isn't pinned. "Regular expressions matched against the object URI" leaves the dialect open (Java java.util.regex? RE2?), and the spec is the only contract between a client writing patterns and whichever service evaluates them.
Since the server evaluates them during a synchronous scan, maybe worth stating whether pattern count/complexity is bounded.
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,283 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, which objects to include, and how often to re-scan. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is managed by Polaris. Dropping it, or writing to it through the Iceberg REST API, breaks the directory:
+the next scan fails or the inventory no longer reflects the objects of the directory. Use the directories API to scan or drop a directory.
+{{< /alert >}}
+
+Polaris provides a simple synchronous scan endpoint but does not schedule scans. External services (e.g. directory table scanning service) can also read the directory configuration through the REST API,
+walk the object store, and write the results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base_location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+| `scan-schedule` | `object` | No | Object representing a scan schedule (trigger, cron, ...). |
+
+### Storage access
+
+Directories use Polaris `StorageAccessConfig` to access the object store.
+
+The `base_location` is validated like a table location when the directory is created and again before each scan:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base_location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base_location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ },
+ "scan-schedule": {
+ "cron": "0 * * * *"
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including filter and scan schedule.
+
+### Drop a directory
+
+**DELETE** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+### Scan a directory
+
+**POST** `/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan`
+
+Lists the objects under the directory `base_location`, applies the configured [filter](#filter) and
+replaces the content of the directory table with the result in a single commit. The response contains
+the number of objects recorded in the table:
+
+```json
+{ "file-count": 42 }
+```
+
+Polaris lists the `base_location` with read-only credentials scoped to that location, and requires the
+`UPDATE_TABLE` privilege on the directory. The scan is synchronous and does not run on the
+`scan-schedule`: it is up to the caller (or an [external service](#external-services-scanning-services))
+to trigger it periodically. If the scan fails, the directory table remains unchanged.
+
+The scan service is disabled by default: it is enabled with the `ENABLE_DIRECTORY_SCAN` feature flag (see
Review Comment:
Only the `scan` is feature-flagged.
Every other optional entity type is gated at the entity level (`ENABLE_GENERIC_TABLES`, `ENABLE_POLICY_STORE`, `ENABLE_SEMANTIC_MODELS`).
Should the whole Directory API be gated while it's beta? The spec also doesn't say what a disabled endpoint returns โ `404`, `501`, `400`?
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: Conflict - The directory already exists under the given namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
Review Comment:
No way to update a directory.
`create/load/drop/scan` but no `update`, so changing filter or scan-schedule means dropping the directory โ which per the docs drops the inventory table and its history. Intended for beta, or is an update endpoint coming in PR 2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4442770114
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
The intent is to align with other Polaris APIs.
I'm fine to change that but it would be probably a bit confusing for users. A possible alternative would be to hook in Generic Tables but I don't think it's a great idea from a semantic perspective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4460514820
##########
site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md:
##########
@@ -300,6 +300,15 @@ Flag to enable or disable the API to reset principal credentials. Defaults to en
---
+##### `polaris.features."ENABLE_DIRECTORIES"`
+
+If true, the directories endpoints are enabled and advertised to clients in the catalog configuration response. If false, the endpoints return 501 Not Implemented. This is a beta feature: the API is under active development and may change in a backward-incompatible way. It is disabled by default; enable it with caution and report any issues encountered.
Review Comment:
This is not used ATM, if we release it, it might be confusing to end users.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,478 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/page-token'
+ - $ref: '#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter`, `scan-schedule` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
Review Comment:
Sorry, but I do not think the IRC spec on `PageToken` is done well:
https://github.com/apache/polaris/blob/5fafc67a832a8900fefb399abc00549d864470d6/spec/iceberg-rest-catalog-open-api.yaml#L2403-L2405
... more points here: https://github.com/apache/polaris/pull/5927#discussion_r4426221501
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4497239557
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
Review Comment:
Can we clarify that the inventory iceberg table provides a snapshot, not necessarily up to date? I think it makes sense to give users the expectation that inventory table may delay.
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
Review Comment:
nit: can we put the `beta` label on the spec yaml?
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its filter.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `filter` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new filter applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). |
+| 5 | `checksum` | `string` | No | The object checksum value computed with the algorithm specified in `checksum_algorithm`. |
+| 6 | `last_modified` | `timestamptz` | No | The last modification timestamp of the object as reported by the object store. |
Review Comment:
Question, not a blocker: Should we make this `required`? I think it's important for sorting, partition at ingestion time. Meanwhile, it could be very useful for consumer, so that they can sort and filter based on ts.
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5167,96 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ Directory:
+ type: object
+ description: |
+ Directory information.
+ - `name` (REQUIRED): name of the directory
+ - `base-location` (REQUIRED): root location of the directory on the object store
+ - `filter` (OPTIONAL): include and exclude patterns for object filtering
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
Review Comment:
IIUC, filter was used for scan the directory and ingesting it to the inventory table. The scanner would need more configurations than a filter. Can we have a dedicated object for scanner configs? We don't have to specify everything now. Filter would be one of field within the ScannerConfig object. Other potential options would be checksum algorithm, freshness, etc. We can figure these options out in followup though.
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its filter.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `filter` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new filter applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). |
+| 5 | `checksum` | `string` | No | The object checksum value computed with the algorithm specified in `checksum_algorithm`. |
+| 6 | `last_modified` | `timestamptz` | No | The last modification timestamp of the object as reported by the object store. |
+| 7 | `metadata` | `map<string, string>` | No | Additional labels and tags from the object store (for example S3 user-defined metadata, storage class, or content encoding). |
+
+Clients (query engines, analytics tools, etc.) interact with the directory table as a regular Iceberg
+table. For instance, you can query it with Spark SQL:
+
+```sql
+SELECT file_uri, size, last_modified
+FROM my_catalog.my_namespace.`product-images__inventory`
+WHERE content_type = 'image/jpeg'
+ AND size > 1048576
+ORDER BY last_modified DESC;
+```
+
+### Credential vending
+
+The directory table lists the objects, it does not give access to them. Reading the directory table returns the
+object URIs and metadata, but a user or engine needs its own access to the object store to read the objects
+themselves.
+
+More precisely:
+
+- When an engine loads the directory table with [credential vending]({{% ref "vended-credentials" %}}), Polaris
+ vends credentials scoped to the directory table location (its data and metadata files), as for any Iceberg
+ table. These credentials are not scoped to the directory `base-location`.
+- Privileges on the directory or on the directory table (for example `TABLE_READ_DATA`) don't grant access to the
+ objects under the `base-location`.
Review Comment:
IIUC, this is going to be phase 2. Should we clarify that? This is planned.
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5167,96 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ Directory:
+ type: object
+ description: |
+ Directory information.
+ - `name` (REQUIRED): name of the directory
+ - `base-location` (REQUIRED): root location of the directory on the object store
+ - `filter` (OPTIONAL): include and exclude patterns for object filtering
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
Review Comment:
it could be optional in case inventory table is missing.
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its filter.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `filter` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new filter applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). |
Review Comment:
Do we want the checksum_algorithm be row-level? In that case, each rows may have a different checksum algorithm. I think a table level config should be good. WDYT?
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5167,96 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ Directory:
+ type: object
+ description: |
+ Directory information.
+ - `name` (REQUIRED): name of the directory
+ - `base-location` (REQUIRED): root location of the directory on the object store
+ - `filter` (OPTIONAL): include and exclude patterns for object filtering
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
Review Comment:
I think we will need to send back inventory table identifier when a directory is loaded. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4500237308
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5167,96 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ Directory:
+ type: object
+ description: |
+ Directory information.
+ - `name` (REQUIRED): name of the directory
+ - `base-location` (REQUIRED): root location of the directory on the object store
+ - `filter` (OPTIONAL): include and exclude patterns for object filtering
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
Review Comment:
Good idea!
I added a `DirectoryScannerConfig` schema, exposed as `scanner-config` in the create and update requests and in `Directory`.
It only contains `filter` for now, we can add other options (`checksum` algorithm, freshness, ...) in follow-ups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4462366113
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,478 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/page-token'
+ - $ref: '#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter`, `scan-schedule` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
Review Comment:
What is not done well with IRC pageToken schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4464968341
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,478 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/page-token'
+ - $ref: '#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter`, `scan-schedule` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
+
+ page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ ScanSchedule:
Review Comment:
The purpose here is informative to be used by the scan services (at their discretion).
The spec doesn't force anything (the scan services can do their own logic), but can clearly define a "good to have" scheduling, up to the scan services to use it or not.
So, this is informative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4478620590
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
Review Comment:
You are right, it was inconsistent with the doc. The intent was to cover the directory `base-location`, which isn't supported yet.
I removed the `Polaris-Directory-Access-Delegation` header and the `storage-access-configs` response field.
Credentials for the inventory table are still vended through the Iceberg REST API, like for any table.
We can add credential vending for the `base-location` in a follow-up on it's designed (I will work on it once this PR is merged).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4500241546
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5167,96 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ Directory:
+ type: object
+ description: |
+ Directory information.
+ - `name` (REQUIRED): name of the directory
+ - `base-location` (REQUIRED): root location of the directory on the object store
+ - `filter` (OPTIONAL): include and exclude patterns for object filtering
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
Review Comment:
Agreed, and I did it.
`Directory` now has an optional `inventory-table` field (`TableIdentifier`), absent if the inventory table doesn't exist. Clients also don't need to rely on the `__inventory` naming convention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4500199916
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
Review Comment:
Yes, no problem ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4522452705
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5193,106 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ DirectoryScannerConfig:
+ type: object
+ description: |
+ Configuration used by the external scanning services to populate the directory inventory table. Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
Review Comment:
Right, clients don't choose the inventory table location for now: Polaris creates the inventory table like any table in the namespace, at the default table location (under the namespace location).
I don't think the inventory table can end up with a different storage configuration than the `base-location`: the directory and its inventory table are in the same namespace, so they resolve the same storage configuration, and the `base-location` is validated against its allowed locations. Credential vending for both will go through the same storage configuration.
I'm not sure about co-locating the inventory table with the `base-location`:
- inside the `base-location`, the scan would list the data and metadata files of the inventory table itself, and writing the inventory table would need write access to the `base-location`, which is often data that should only be read (for example a bucket written by another team or an ingestion pipeline).
- next to the `base-location` (same parent), the parent may be outside the allowed locations (or not exist, for a bucket root), and the inventory table could collide with other data.
The default table location avoids these issues. If needed, we can add an optional inventory table location to the create request later, validated like a table location.
WDYT?
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Load a directory from the catalog under the given namespace.
+
+ The response contains the directory configuration and, when the directory inventory table exists,
+ its identifier.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Update the mutable configuration of a directory (`scanner-config` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new scanner configuration applies from the next scan.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ type: string
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ schemas:
+ DirectoryPageToken:
+ description: >
+ An opaque token used to paginate the list of directories. The server returns it as
+ `next-page-token` when more results may be available, and the client passes it unchanged as the
+ `pageToken` query parameter to fetch the next page. The client does not need to send a
+ `pageToken` to start paginating: any response may be paginated, whether or not the request
+ carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no
+ more results.
+ type: [ string, "null" ]
+
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ DirectoryScannerConfig:
+ type: object
+ description: >
+ Configuration used by the external scanning services to populate the directory inventory table.
+ Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ UpdateDirectoryRequest:
+ type: object
+ description: >
+ Fields to update on a directory. Omitted fields are left unchanged.
+ properties:
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ Directory:
+ type: object
+ description: >
+ Directory information.
+
+ - `name` (REQUIRED): name of the directory
+
+ - `base-location` (REQUIRED): root location of the directory on the object store
+
+ - `scanner-config` (OPTIONAL): configuration for the scanning services, such as the include and exclude
+ patterns for object filtering
+
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+
+ - `inventory-table` (OPTIONAL): identifier of the directory inventory table, assigned by Polaris. It is
+ absent if the inventory table does not exist, for example if it was dropped through the Iceberg REST API.
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ inventory-table:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/TableIdentifier'
+
+ LoadDirectoryResponse:
+ description: Result used when a directory is successfully loaded.
+ type: object
+ required:
+ - directory
+ properties:
+ directory:
+ $ref: '#/components/schemas/Directory'
+
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/TableIdentifier'
+
+ responses:
+
+ LoadDirectoryResponse:
+ description: Directory result if successfully loaded a directory.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/LoadDirectoryResponse'
+
+ CreateDirectoryResponse:
+ description: Directory result if successfully created a directory.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/LoadDirectoryResponse'
+
+ ListDirectoriesResponse:
Review Comment:
Which wrapper do you mean?
- `CreateDirectoryResponse` in `components/responses`, which is the same as `LoadDirectoryResponse` (`updateDirectory` already references `LoadDirectoryResponse` directly)? I can remove it.
- or the `{"directory": ...}` object of `LoadDirectoryResponse`? I would keep this one: it follows `LoadGenericTableResponse`, and leaves room to return the storage credentials for the `base-location` next to the directory in phase 2 without breaking clients.
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,272 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+The directory table is a snapshot of the objects as of the last scan, not a live view of the object store. Objects
+added, changed, or removed after the last scan appear in the directory table only after the next scan, so the
+directory table can lag behind the object store.
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `scanner-config` | `object` | No | Configuration for the scanning services, such as the filter that controls which objects are added to the directory table during a scan. See [Scanner configuration](#scanner-configuration). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Scanner configuration
+
+The `scanner-config` object holds the configuration used by the
+[external scanning services](#external-services-scanning-services) to populate the directory table. Polaris stores
+it and returns it when the directory is loaded, but does not act on it.
+
+It contains a [`filter`](#filter) that controls which objects are added to the directory table during a scan.
+
+#### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "scanner-config": {
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "scanner-config": {
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its scanner configuration, and the
+identifier of its [directory table](#directory-table) in the `inventory-table` field. The `inventory-table` field
+is absent if the directory table does not exist.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `scanner-config` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new scanner configuration applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table identifier is also returned in the `inventory-table`
+field when the directory is loaded. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). It is set per object because objects under the same location can have checksums computed with different algorithms (for example S3 objects uploaded with different checksum settings), so a scanning service can record the checksum reported by the object store without reading the object. |
+| 5 | `checksum` | `string` | No | The object checksum value computed with the algorithm specified in `checksum_algorithm`. |
+| 6 | `last_modified` | `timestamptz` | Yes | The last modification timestamp of the object as reported by the object store. |
+| 7 | `metadata` | `map<string, string>` | No | Additional labels and tags from the object store (for example S3 user-defined metadata, storage class, or content encoding). |
Review Comment:
Good point. I added the inventory table schema to the spec, in the `Directory` description, with the compatibility rule: the fields are not renamed or removed, and new fields are only added as optional fields. The doc says the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "sririshindra (via GitHub) via issues" <is...@polaris.apache.org>.
sririshindra commented on PR #5927:
URL: https://github.com/apache/polaris/pull/5927#issuecomment-6756520639
FYI: following the pagination discussion here and on #5943, I opened #6028 (draft). It adds shared pagination definitions for Polaris APIs in `spec/polaris-pagination.yaml`: the `pageToken`/`pageSize` query parameters and a page token schema, with the same rules as `directory-page-token` and `DirectoryPageToken` here. Mentioning it in case it's useful for the Directories API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4462350355
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,478 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/page-token'
+ - $ref: '#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter`, `scan-schedule` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
+
+ page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ ScanSchedule:
Review Comment:
Do we need to expose the scan schedule in the spec? I think the scheduling mechanism should be abstracted away from users. For example, scans could be triggered by cron, S3 notifications, or other mechanisms. We probably shouldn't prescribe a specific scheduling mechanism in the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on PR #5927:
URL: https://github.com/apache/polaris/pull/5927#issuecomment-6615547696
from CI: `ERROR: bundled OpenAPI spec is out of date. Please run 'make spec-bundle' and commit the changes.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4464981202
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,478 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/page-token'
+ - $ref: '#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter`, `scan-schedule` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
+
+ page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ ScanSchedule:
Review Comment:
I can remove this part, we can re-add it later "adhoc". Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4464975784
##########
site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md:
##########
@@ -300,6 +300,15 @@ Flag to enable or disable the API to reset principal credentials. Defaults to en
---
+##### `polaris.features."ENABLE_DIRECTORIES"`
+
+If true, the directories endpoints are enabled and advertised to clients in the catalog configuration response. If false, the endpoints return 501 Not Implemented. This is a beta feature: the API is under active development and may change in a backward-incompatible way. It is disabled by default; enable it with caution and report any issues encountered.
Review Comment:
Are you suggesting to move the documentation in the second directory PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4464981202
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,478 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/page-token'
+ - $ref: '#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter`, `scan-schedule` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
+
+ page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ ScanSchedule:
Review Comment:
I can remove this part, we can re-add it later "adhoc".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4426221501
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
Review Comment:
I disagree with how pagination is defined in IRC, specifically:
https://github.com/apache/polaris/blob/5fafc67a832a8900fefb399abc00549d864470d6/spec/iceberg-rest-catalog-open-api.yaml#L2403-L2405
The idea of clients forcing servers into returning all data is against the very premise of pagination.
We are fixing this in our IRC API impl. in #5282 and #5553. I think we should do proper pagination in the Directories API upfront:
* Clients should always expect that a response may be paginated
* Clients do not need to set either the `page-token` or `page-size` on the initial request to initiate pagination. The server decides.
* The server decides on actual page sizes. The can be smaller than the `page-size` hint.
* A server may choose not to support pagination and return all data in one chunk (but Polaris will support pagination, I assume).
Related discussions:
https://lists.apache.org/thread/k81ptyktdbdf8gynncgk3o04mqt85zyk
https://lists.apache.org/thread/mmqllcbt2lfsrq571cfwbymgfwoxqdf0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "ayushtkn (via GitHub) via issues" <is...@polaris.apache.org>.
ayushtkn commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4430879865
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
Review Comment:
+1 on doing proper pagination in the Directories API from the start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4442608261
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
I think it's semantically different: my take on Generic Table is for non-Iceberg table.
As Inventory is a Iceberg table and Directory is a new "feature", I think it makes more sense to have dedicated endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4462312936
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
I think a directory table is essentially a type of generic table. They share the same location management requirements, including but not limited to base location, overlap checking, and credential vending.
This makes directory tables quite different from other entities, such as policies and tags, which don't require storage.
I'm not sure we need to reinvent what we've already implemented for generic tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4522451442
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
Review Comment:
Agreed. #6028 adds shared pagination definitions for the Polaris APIs (`spec/polaris-pagination.yaml`), with the same rules as `directory-page-token` and `DirectoryPageToken` here. Once it lands, I will switch the Directories API to the shared definitions in a follow-up.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Load a directory from the catalog under the given namespace.
+
+ The response contains the directory configuration and, when the directory inventory table exists,
+ its identifier.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Update the mutable configuration of a directory (`scanner-config` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new scanner configuration applies from the next scan.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Remove a directory under the given namespace from the catalog
Review Comment:
Dropping a directory also drops its inventory table, without purge (like dropping an Iceberg table with `purgeRequested=false`): the data and metadata files of the inventory table are kept. The objects under the `base-location` are never deleted: Polaris doesn't own them, the scanning services only list them.
I clarified both in the `dropDirectory` description and in the doc. If we want to purge the inventory table files, we can add an optional `purgeRequested` query parameter in a follow-up, as for Iceberg tables.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
Review Comment:
Yes, the inventory table is always created with the directory, empty, in the same namespace, and the create response returns its identifier in `inventory-table`. Creating a directory doesn't trigger a scan: Polaris doesn't scan, the external scanning services populate the inventory table. I clarified this in the `createDirectory` description and in the doc.
I would not add an option to skip the inventory table now. The inventory table is what makes the objects discoverable through Iceberg. Without it, a directory is only a registered location, as Polaris doesn't vend credentials for the `base-location` yet. As `inventory-table` is already optional in `Directory`, we can add this option later (for the managed volume use case) without breaking clients.
WDYT?
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5193,106 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ DirectoryScannerConfig:
+ type: object
+ description: |
+ Configuration used by the external scanning services to populate the directory inventory table. Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
Review Comment:
I would keep a single `base-location` for now:
- a directory maps one root to one inventory table. The `base-location` validation (allowed locations, no overlap with sibling tables or directories), the scan, and the credential vending planned for phase 2 all work per location.
- several roots can be several directories, each with its own inventory table (and a view on top of them if needed).
- generic tables also have a single `base-location` (the spec says a table with multiple disjoint locations is not supported).
The tradeoff: if we add more locations later, a scanner that doesn't know the new field would only scan the `base-location` and miss objects. As the API is beta, I think it's acceptable, but I'm open to adding an optional `additional-locations` now if you prefer.
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5193,106 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ DirectoryScannerConfig:
+ type: object
+ description: |
+ Configuration used by the external scanning services to populate the directory inventory table. Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
Review Comment:
No, the `base-location` is validated like a table location: it must be within the catalog's allowed locations, and within the namespace location unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`. It was described in the "Base location" section of the doc, and I added it to the `base-location` description in the spec too.
And yes, it's managed by the storage configuration, resolved from the directory's namespace path as for a table in the same namespace. The scan endpoint in #5928 already uses it to get read and list credentials for the `base-location`, and the credential vending in phase 2 will use it too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4426221501
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
Review Comment:
I disagree with how pagination is defined in IRC, specifically:
https://github.com/apache/polaris/blob/5fafc67a832a8900fefb399abc00549d864470d6/spec/iceberg-rest-catalog-open-api.yaml#L2403-L2405
The idea of clients forcing servers into returning all data is against the very premise of pagination.
We are fixing this in our IRC API impl. in #5553. I think we should do proper pagination in the Directories API upfront:
* Clients should always expect that a response may be paginated
* Clients do not need to set either the `page-token` or `page-size` on the initial request to initiate pagination. The server decides.
* The server decides on actual page sizes. The can be smaller than the `page-size` hint.
* A server may choose not to support pagination and return all data in one chunk (but Polaris will support pagination, I assume).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4442745903
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: Conflict - The directory already exists under the given namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
Review Comment:
Good catch. Added an `updateDirectory` endpoint (`POST .../directories/{directory}`) so `filter`, `scan-schedule` and `properties` can be changed without dropping the directory; `name` and `base-location` stay immutable, and the inventory table and its history are preserved. Spec and docs updated in 225935edbd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4462787987
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,478 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/page-token'
+ - $ref: '#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter`, `scan-schedule` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
Review Comment:
@flyrain : please see linked comment (above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4474777798
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
Review Comment:
Which location do these credentials cover: the directoryโs `base-location` or the inventory table? `directories.md` says vending credentials for the source objects isnโt supported yet, and inventory-table credentials come from the Iceberg API. Iโm not sure how this header fits with that. If directory credential vending is deferred, do we need to expose the header now?
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryPageToken:
+ description: >
+ An opaque token used to paginate the list of directories. The server returns it as
+ `next-page-token` when more results may be available, and the client passes it unchanged as the
+ `pageToken` query parameter to fetch the next page. The client does not need to send a
+ `pageToken` to start paginating: any response may be paginated, whether or not the request
+ carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no
+ more results.
+ type: [ string, "null" ]
+
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ UpdateDirectoryRequest:
+ type: object
+ description: >
+ Fields to update on a directory. Omitted fields are left unchanged.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ Directory:
+ type: object
+ description: >
+ Directory information.
+
+ - `name` (REQUIRED): name of the directory
+
+ - `base-location` (REQUIRED): root location of the directory on the object store
+
+ - `filter` (OPTIONAL): include and exclude patterns for object filtering
+
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ LoadDirectoryResponse:
+ description: Result used when a directory is successfully loaded.
+ type: object
+ required:
+ - directory
+ properties:
+ directory:
+ $ref: '#/components/schemas/Directory'
+ storage-access-configs:
+ type: array
+ description: |
+ Storage configurations required to access the directory, encompassing both credentials and other necessary
+ settings for storage access, such as the client region for S3.
+ items:
+ $ref: './common-schemas.yaml#/components/schemas/StorageAccessConfig'
+
+ ScanDirectoryResponse:
+ description: Result of a directory scan.
+ type: object
+ required:
+ - file-count
+ properties:
+ file-count:
+ type: integer
+ format: int64
+ description: Number of objects recorded in the directory inventory table after the scan.
Review Comment:
I'm not sure this should be a sync operation. A scan may take a long time. Async mode will be preferrabe.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
Review Comment:
I'm not sure if we need a scan endpoint. Could you clarify the use cases for a synchronous scan endpoint? An explicit refresh could be useful for initial population or reconciliation, but those operations can also run asynchronously. Since the proposal(https://github.com/apache/polaris/blob/9a362ec9865777138ddf8043c7a4968bad0cc53d/site/content/in-dev/unreleased/directories.md#external-services-scanning-services) supports external event-driven or scheduled scanners, how does this endpoint coordinate with them when they update the same inventory? Iโd prefer to establish the execution and ownership model first, then decide whether we need a public scan operation and how callers observe its completion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4519963594
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Load a directory from the catalog under the given namespace.
+
+ The response contains the directory configuration and, when the directory inventory table exists,
+ its identifier.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Update the mutable configuration of a directory (`scanner-config` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new scanner configuration applies from the next scan.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Remove a directory under the given namespace from the catalog
Review Comment:
Do we remove the inventory table? Do we purge all files?
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5193,106 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ DirectoryScannerConfig:
+ type: object
+ description: |
+ Configuration used by the external scanning services to populate the directory inventory table. Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ scanner-config:
Review Comment:
+1. thanks for adding it.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
Review Comment:
Do we create the inventory table by default? If not, how do we return the inventory table identifier? If yes, does it involve the first-time scan?
Can clients control the behavior(whether to create the inventory table)? For example, a user may just need a simple managed volume without the inventory table, which is a reasonable.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
Review Comment:
Not a blocker: I'd suggest to use one common pagination schema cross Polaris end points, if we are not using IRC pagination schema. We now have different pagination schema in different end points. This fragmentation is unnecessary. I'm OK with a followup. Can we file an issue for this.
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,272 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+The directory table is a snapshot of the objects as of the last scan, not a live view of the object store. Objects
+added, changed, or removed after the last scan appear in the directory table only after the next scan, so the
+directory table can lag behind the object store.
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `scanner-config` | `object` | No | Configuration for the scanning services, such as the filter that controls which objects are added to the directory table during a scan. See [Scanner configuration](#scanner-configuration). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Scanner configuration
+
+The `scanner-config` object holds the configuration used by the
+[external scanning services](#external-services-scanning-services) to populate the directory table. Polaris stores
+it and returns it when the directory is loaded, but does not act on it.
+
+It contains a [`filter`](#filter) that controls which objects are added to the directory table during a scan.
+
+#### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "scanner-config": {
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "scanner-config": {
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its scanner configuration, and the
+identifier of its [directory table](#directory-table) in the `inventory-table` field. The `inventory-table` field
+is absent if the directory table does not exist.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `scanner-config` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new scanner configuration applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table identifier is also returned in the `inventory-table`
+field when the directory is loaded. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). It is set per object because objects under the same location can have checksums computed with different algorithms (for example S3 objects uploaded with different checksum settings), so a scanning service can record the checksum reported by the object store without reading the object. |
+| 5 | `checksum` | `string` | No | The object checksum value computed with the algorithm specified in `checksum_algorithm`. |
+| 6 | `last_modified` | `timestamptz` | Yes | The last modification timestamp of the object as reported by the object store. |
+| 7 | `metadata` | `map<string, string>` | No | Additional labels and tags from the object store (for example S3 user-defined metadata, storage class, or content encoding). |
Review Comment:
Should we put these schema as part of the spec? I think the clients would expect a stable schema. Any schema changes may break existing clients.
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5193,106 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ DirectoryScannerConfig:
+ type: object
+ description: |
+ Configuration used by the external scanning services to populate the directory inventory table. Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
Review Comment:
Do we want to support multiple locations? We have debated a lot when generic table was introduced. I think supporting multiple locations would be more extensible even most use cases are with a single location. However, I could be convinced otherwise. This is to make sure we don't constrain ourselves by design, the same reason we design generic tables with multiple locations.
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5193,106 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ DirectoryScannerConfig:
+ type: object
+ description: |
+ Configuration used by the external scanning services to populate the directory inventory table. Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
Review Comment:
Does it mean we never allow clients to specify the inventory table location?
In that case, it has to locate under the ns where the directory table exists, which becomes tricky when we allow the `base-location` be outside of the current namespaces, since it may select a different storage configurations. For example, we may have to vend credential for aws s3 and azure when it comes to the phase we vend credentials for both directory table and its inventory table.
If we allow clients to specify location, it actually hits the same problem.
I think a way to simplify the situation is to let the inventory table always co-locate with the directory `base-location`, the same parents is probably the easiest thing to do. WDTY?
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Load a directory from the catalog under the given namespace.
+
+ The response contains the directory configuration and, when the directory inventory table exists,
+ its identifier.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Update the mutable configuration of a directory (`scanner-config` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new scanner configuration applies from the next scan.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ type: string
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ schemas:
+ DirectoryPageToken:
+ description: >
+ An opaque token used to paginate the list of directories. The server returns it as
+ `next-page-token` when more results may be available, and the client passes it unchanged as the
+ `pageToken` query parameter to fetch the next page. The client does not need to send a
+ `pageToken` to start paginating: any response may be paginated, whether or not the request
+ carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no
+ more results.
+ type: [ string, "null" ]
+
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ DirectoryScannerConfig:
+ type: object
+ description: >
+ Configuration used by the external scanning services to populate the directory inventory table.
+ Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ UpdateDirectoryRequest:
+ type: object
+ description: >
+ Fields to update on a directory. Omitted fields are left unchanged.
+ properties:
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ Directory:
+ type: object
+ description: >
+ Directory information.
+
+ - `name` (REQUIRED): name of the directory
+
+ - `base-location` (REQUIRED): root location of the directory on the object store
+
+ - `scanner-config` (OPTIONAL): configuration for the scanning services, such as the include and exclude
+ patterns for object filtering
+
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+
+ - `inventory-table` (OPTIONAL): identifier of the directory inventory table, assigned by Polaris. It is
+ absent if the inventory table does not exist, for example if it was dropped through the Iceberg REST API.
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ base-location:
+ type: string
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+ inventory-table:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/TableIdentifier'
+
+ LoadDirectoryResponse:
+ description: Result used when a directory is successfully loaded.
+ type: object
+ required:
+ - directory
+ properties:
+ directory:
+ $ref: '#/components/schemas/Directory'
+
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/TableIdentifier'
+
+ responses:
+
+ LoadDirectoryResponse:
+ description: Directory result if successfully loaded a directory.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/LoadDirectoryResponse'
+
+ CreateDirectoryResponse:
+ description: Directory result if successfully created a directory.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/LoadDirectoryResponse'
+
+ ListDirectoriesResponse:
Review Comment:
I think this wrapper is not needed.
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its filter.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `filter` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new filter applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). |
Review Comment:
Makes sense. Let's keep it per row.
##########
spec/generated/bundled-polaris-catalog-service.yaml:
##########
@@ -4987,6 +5193,106 @@ components:
settings for storage access, such as the client region for S3.
items:
$ref: '#/components/schemas/StorageAccessConfig'
+ DirectoryPageToken:
+ description: |
+ An opaque token used to paginate the list of directories. The server returns it as `next-page-token` when more results may be available, and the client passes it unchanged as the `pageToken` query parameter to fetch the next page. The client does not need to send a `pageToken` to start paginating: any response may be paginated, whether or not the request carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no more results.
+ type:
+ - string
+ - 'null'
+ ListDirectoriesResponse:
+ type: object
+ properties:
+ next-page-token:
+ $ref: '#/components/schemas/DirectoryPageToken'
+ identifiers:
+ type: array
+ uniqueItems: true
+ items:
+ $ref: '#/components/schemas/TableIdentifier'
+ DirectoryFilter:
+ type: object
+ description: |
+ Controls which objects are included in the directory table during a scan. If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+ DirectoryScannerConfig:
+ type: object
+ description: |
+ Configuration used by the external scanning services to populate the directory inventory table. Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
Review Comment:
Do we allow the directory `base-location` be outside of the namespace location, or catalog's allowed locations? Is this location also managed by storage configuration? I assume we have to, otherwise, the credential vending(phase 2) wouldn't be possible. WDYT?
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,411 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Load a directory from the catalog under the given namespace.
+
+ The response contains the directory configuration and, when the directory inventory table exists,
+ its identifier.
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Update the mutable configuration of a directory (`scanner-config` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new scanner configuration applies from the next scan.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: |
+ **Beta:** The Directories API is under active development and may change in a
+ backward-incompatible way.
+
+ Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ type: string
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ schemas:
+ DirectoryPageToken:
+ description: >
+ An opaque token used to paginate the list of directories. The server returns it as
+ `next-page-token` when more results may be available, and the client passes it unchanged as the
+ `pageToken` query parameter to fetch the next page. The client does not need to send a
+ `pageToken` to start paginating: any response may be paginated, whether or not the request
+ carried a `pageToken` or a `pageSize`. A `null` or missing `next-page-token` means there are no
+ more results.
+ type: [ string, "null" ]
+
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base-location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
+ exclude:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions (`java.util.regex` syntax) matched against the object URI. An object matching any pattern is excluded, even if it also matches an include pattern.
+
+ DirectoryScannerConfig:
+ type: object
+ description: >
+ Configuration used by the external scanning services to populate the directory inventory table.
+ Polaris stores it and returns it when the directory is loaded, but does not act on it.
+ properties:
+ filter:
+ $ref: '#/components/schemas/DirectoryFilter'
+
+ CreateDirectoryRequest:
+ type: object
+ required:
+ - name
+ - base-location
+ properties:
+ name:
+ type: string
+ description: The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table.
+ base-location:
+ type: string
+ description: The external object store location to use as a root for scanning objects (e.g. s3://bucket/path/). This is where the scanned objects live, not the location of the Iceberg inventory table, which is assigned by Polaris.
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ UpdateDirectoryRequest:
+ type: object
+ description: >
+ Fields to update on a directory. Omitted fields are left unchanged.
+ properties:
+ scanner-config:
+ $ref: '#/components/schemas/DirectoryScannerConfig'
+ properties:
+ type: object
+ additionalProperties:
+ type: string
+
+ Directory:
+ type: object
+ description: >
+ Directory information.
+
+ - `name` (REQUIRED): name of the directory
+
+ - `base-location` (REQUIRED): root location of the directory on the object store
+
+ - `scanner-config` (OPTIONAL): configuration for the scanning services, such as the include and exclude
+ patterns for object filtering
+
+ - `properties` (OPTIONAL): properties for the directory passed on creation
+
+ - `inventory-table` (OPTIONAL): identifier of the directory inventory table, assigned by Polaris. It is
Review Comment:
+1 on optional inventory-table!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4426221501
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
Review Comment:
I disagree with how pagination is defined in IRC, specifically:
https://github.com/apache/polaris/blob/5fafc67a832a8900fefb399abc00549d864470d6/spec/iceberg-rest-catalog-open-api.yaml#L2403-L2405
The idea of clients forcing servers into returning all data is against the very premise of pagination.
We are fixing this in the IRC API in #5553. I think we should do proper pagination in the Directories API upfront:
* Clients should always expect that a response may be paginated
* Clients do not need to set either the `page-token` or `page-size` on the initial request to initiate pagination. The server decides.
* The server decides on actual page sizes. The can be smaller than the `page-size` hint.
* A server may choose not to support pagination and return all data in one chunk (but Polaris will support pagination, I assume).
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: Conflict - The directory already exists under the given namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
Review Comment:
non-blocking: I do not think yaml cross-references are convenience for downstream users. If someone wants to copy this API definition to generate clients, the IRC spec has to be copied too, but the IRC spec does not appear to be a conceptual prerequisite to the Directories spec.
Also (perhaps more critical): changes to IRC spec do not have to automatically trickle down to the directories spec, IMHO. It may be a weak point with respect to these two parameters, but in general I think it makes sense ๐ค
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
Non-blocking: this path prefix interleaves with other Polaris API (e.g. Generic Tables, Policies...). Since endpoints are distinct, it is not a problem. However, it makes separating APIs more difficult.
The `v1` prefix is shared and it is usually perceived as the "API root".
There is no static prefix just for the directories API, which can complicate network routing if someone wants to segregate traffic by API.
I know this fit the existing pattern, so my comment is non-blocking ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4442755789
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: Return all directory identifiers under this namespace
+ operationId: listDirectories
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-token'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: Conflict - The directory already exists under the given namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base_location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
+ description: >
+ Optional header indicating that the client supports delegated access
+ using a specific mechanism. If the header is present, the server must
+ either provide access using the requested mechanism or fail the request.
+ If `vended-credentials` is specified, the server must return scoped
+ storage credentials in the `storage-access-configs` field of the response.
+ required: false
+ schema:
+ type: array
+ items:
+ type: string
+ enum:
+ - vended-credentials
+ style: simple
+ explode: false
+ example: "vended-credentials"
+
+ schemas:
+ DirectoryFilter:
+ type: object
+ description: >
+ Controls which objects are included in the directory table during a scan.
+ If no filter is set, all objects found under the directory base_location are included.
+ properties:
+ include:
+ type: array
+ items:
+ type: string
+ description: A list of regular expressions matched against the object URI. An object is included if it matches at least one pattern (or if `include` is omitted).
Review Comment:
That's intentional as the scan service can use the regex of his choice. Here's a "storage" not format enforcement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4465152910
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,478 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/page-token'
+ - $ref: '#/components/parameters/page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter`, `scan-schedule` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/PageToken'
Review Comment:
I use a local impl for now.
##########
site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md:
##########
@@ -300,6 +300,15 @@ Flag to enable or disable the API to reset principal credentials. Defaults to en
---
+##### `polaris.features."ENABLE_DIRECTORIES"`
+
+If true, the directories endpoints are enabled and advertised to clients in the catalog configuration response. If false, the endpoints return 501 Not Implemented. This is a beta feature: the API is under active development and may change in a backward-incompatible way. It is disabled by default; enable it with caution and report any issues encountered.
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4469721270
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
Review Comment:
optional: Header matches the existing IRC pattern, so it is fine.
However, in general, this parameter feel like a first class parameter in the API. It affects what goes into the response payload. It might be worth considering putting it on the request payload or query parameter.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
Review Comment:
optional: Header matches the existing IRC pattern, so it is fine.
However, in general, this parameter feels like a first class parameter in the API. It affects what goes into the response payload. It might be worth considering putting it on the request payload or query parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4469759402
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
Review Comment:
nit: `directory-page-size` has an inline schema... I guess we might as well have inline schema for `directory-page-token`.
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,471 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+
+ get:
+ tags:
+ - Directory API
+ summary: List all directory identifiers underneath a given namespace
+ description: >
+ Return the directory identifiers under this namespace. The identifiers are those of the directories
+ (for example `product-images`), not of their `__inventory` tables.
+
+
+ The response may be paginated: clients should always check `next-page-token` and, when present,
+ pass it as `pageToken` to fetch the next page. Neither `pageToken` nor `pageSize` is required on the
+ first request. The server decides the actual page size, which may be smaller or larger than the
+ `pageSize` hint, and may return all results in a single response.
+ operationId: listDirectories
+ parameters:
+ - $ref: '#/components/parameters/directory-page-token'
+ - $ref: '#/components/parameters/directory-page-size'
+ responses:
+ 200:
+ $ref: '#/components/responses/ListDirectoriesResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Create a directory under the given namespace
+ description:
+ Create a directory under the given namespace, and return the created directory information as a response.
+ operationId: createDirectory
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/CreateDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description: Not Found - The namespace specified does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ NamespaceNotFound:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchNamespaceError'
+ 409:
+ description: >
+ Conflict - The directory already exists under the given namespace, or its `<name>__inventory`
+ table name is already used by another table or directory in the namespace
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ TableAlreadyExists:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/TableAlreadyExistsError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ get:
+ tags:
+ - Directory API
+ summary: Load a directory under the given namespace from the catalog
+ operationId: loadDirectory
+ description:
+ Load a directory from the catalog under the given namespace.
+
+ The response contains all directory information passed during create.
+
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to load does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToLoadDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ post:
+ tags:
+ - Directory API
+ summary: Update a directory under the given namespace
+ operationId: updateDirectory
+ description: >
+ Update the mutable configuration of a directory (`filter` and `properties`).
+ Only the fields present in the request are changed, and a provided field replaces the existing value.
+ The `name` and `base-location` cannot be changed. The directory inventory table and its
+ history are preserved. The new filter applies from the next scan.
+ parameters:
+ - $ref: '#/components/parameters/polaris-directory-data-access'
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/UpdateDirectoryRequest'
+ responses:
+ 200:
+ $ref: '#/components/responses/LoadDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to update does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToUpdateDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ delete:
+ tags:
+ - Directory API
+ summary: Drop a directory under the given namespace from the catalog
+ operationId: dropDirectory
+ description: Remove a directory under the given namespace from the catalog
+ responses:
+ 204:
+ description: Success, no content
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to drop does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToDeleteDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}/scan:
+ parameters:
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/prefix'
+ - $ref: '../iceberg-rest-catalog-open-api.yaml#/components/parameters/namespace'
+ - $ref: '#/components/parameters/directory'
+
+ post:
+ tags:
+ - Directory API
+ summary: Scan a directory and refresh its inventory table
+ operationId: scanDirectory
+ description: >
+ Lists the objects under the directory `base-location`, applies the configured filter
+ and replaces the content of the directory inventory table with the result in a single
+ commit. If the scan fails, the inventory table is left unchanged.
+ responses:
+ 200:
+ $ref: '#/components/responses/ScanDirectoryResponse'
+ 400:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/BadRequestErrorResponse'
+ 401:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/UnauthorizedResponse'
+ 403:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ForbiddenResponse'
+ 404:
+ description:
+ Not Found - NoSuchTableError, directory to scan does not exist
+ content:
+ application/json:
+ schema:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'
+ examples:
+ DirectoryToScanDoesNotExist:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/examples/NoSuchTableError'
+ 503:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServiceUnavailableResponse'
+ 5XX:
+ $ref: '../iceberg-rest-catalog-open-api.yaml#/components/responses/ServerErrorResponse'
+
+components:
+ parameters:
+ directory-page-token:
+ name: pageToken
+ in: query
+ description: Opaque token returned as `next-page-token` by a previous response. Omit it to start listing.
+ required: false
+ schema:
+ $ref: '#/components/schemas/DirectoryPageToken'
+
+ directory-page-size:
+ name: pageSize
+ in: query
+ description: >
+ A hint for the maximum number of results per page. The server decides the actual page size and
+ may return fewer or more results, or all of them at once.
+ required: false
+ schema:
+ type: integer
+ minimum: 1
+
+ directory:
+ name: directory
+ in: path
+ description: A directory name
+ required: true
+ schema:
+ type: string
+ example: "product-images"
+
+ polaris-directory-data-access:
+ name: Polaris-Directory-Access-Delegation
+ in: header
Review Comment:
optional: Header matches the existing IRC patter, so it is fine.
However, in general, this parameter feel like a first class parameter in the API. It affects what goes into the response payload. It might be worth considering putting it on the request payload or query parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4428584768
##########
spec/polaris-catalog-apis/directories-api.yaml:
##########
@@ -0,0 +1,393 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+---
+paths:
+ /polaris/v1/{prefix}/namespaces/{namespace}/directories:
Review Comment:
Do we need a new endpoint for the directory? It is a sub type of generic table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4500231828
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its filter.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `filter` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new filter applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). |
+| 5 | `checksum` | `string` | No | The object checksum value computed with the algorithm specified in `checksum_algorithm`. |
+| 6 | `last_modified` | `timestamptz` | No | The last modification timestamp of the object as reported by the object store. |
+| 7 | `metadata` | `map<string, string>` | No | Additional labels and tags from the object store (for example S3 user-defined metadata, storage class, or content encoding). |
+
+Clients (query engines, analytics tools, etc.) interact with the directory table as a regular Iceberg
+table. For instance, you can query it with Spark SQL:
+
+```sql
+SELECT file_uri, size, last_modified
+FROM my_catalog.my_namespace.`product-images__inventory`
+WHERE content_type = 'image/jpeg'
+ AND size > 1048576
+ORDER BY last_modified DESC;
+```
+
+### Credential vending
+
+The directory table lists the objects, it does not give access to them. Reading the directory table returns the
+object URIs and metadata, but a user or engine needs its own access to the object store to read the objects
+themselves.
+
+More precisely:
+
+- When an engine loads the directory table with [credential vending]({{% ref "vended-credentials" %}}), Polaris
+ vends credentials scoped to the directory table location (its data and metadata files), as for any Iceberg
+ table. These credentials are not scoped to the directory `base-location`.
+- Privileges on the directory or on the directory table (for example `TABLE_READ_DATA`) don't grant access to the
+ objects under the `base-location`.
Review Comment:
Yes, I clarified that credential vending for the objects under the `base-location` is not supported yet and is planned for a future release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4500227777
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its filter.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `filter` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new filter applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). |
+| 5 | `checksum` | `string` | No | The object checksum value computed with the algorithm specified in `checksum_algorithm`. |
+| 6 | `last_modified` | `timestamptz` | No | The last modification timestamp of the object as reported by the object store. |
Review Comment:
Agreed, and done ๐
All the object stores we target report a last-modified time when listing. Starting with `required` also keeps our options open: Iceberg allows making a required column optional later, but not the reverse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the Directories REST API specification and documentation [polaris]
Posted by "jbonofre (via GitHub) via issues" <is...@polaris.apache.org>.
jbonofre commented on code in PR #5927:
URL: https://github.com/apache/polaris/pull/5927#discussion_r4500223529
##########
site/content/in-dev/unreleased/directories.md:
##########
@@ -0,0 +1,251 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Directories
+type: docs
+weight: 450
+---
+
+## Overview
+
+{{< alert important >}}
+Directories are in beta. Please use it with caution and report any issue if encountered.
+{{< /alert >}}
+
+Directories make objects (including unstructured data like images, videos, documents, and other objects) discoverable alongside
+structured Iceberg tables within a Polaris catalog.
+A directory points to a base location/prefix on an object store and automatically tracks the objects it contains by maintaining
+an Iceberg table with object-level metadata such as URI, size, content type, checksum, ...
+
+This means query engines and tools that already know how to read Iceberg tables can discover and
+access unstructured data with little or no extra work (accessing the object itself).
+
+## Concepts
+
+A directory has two main parts:
+
+1. **Directory configuration** โ stored by the Polaris server. It describes _where_ the data lives,
+ how to authenticate, and which objects to include. The configuration "lives" in a namespace.
+2. **Directory table** โ an Iceberg table serving as the inventory of all objects contained in the directory, one row per object discovered during a scan.
+ The directory table is named after the configuration, with the `__inventory` suffix (for example `product-images__inventory`), because a directory and a table cannot share a name in a namespace.
+
+{{< alert warning >}}
+The directory table is created and dropped with its directory. Dropping it through the Iceberg REST API breaks the
+directory: use the directories API to drop a directory.
+{{< /alert >}}
+
+Polaris does not scan directories itself. [External services](#external-services-scanning-services) (e.g. directory
+table scanning service) read the directory configuration through the REST API, walk the object store, and write the
+results into the directory table.
+
+## Configuration
+
+A directory is described by the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | `string` | Yes | The name of the directory. It is used in REST endpoint paths and is used, followed by the `__inventory` suffix, as the name of the corresponding Iceberg table. |
+| `base-location` | `string` | Yes | The external object store location to use as a root for scanning objects; this is not the location of the Iceberg inventory table, which is assigned by Polaris (for example `s3://my-bucket/images/`, `gs://my-bucket/docs/`, or `file:///data/local/`). |
+| `filter` | `object` | No | Include and exclude patterns that control which objects are added to the directory table during a scan. See [Filter](#filter). |
+
+### Base location
+
+The `base-location` is validated like a table location when the directory is created:
+it must be within the catalog's allowed locations. As for tables, it must also be within the namespace location
+unless the catalog sets `polaris.config.allow.unstructured.table.location` to `true`.
+
+### Filter
+
+The `filter` object controls which objects are included in the directory table during a scan.
+
+If no filter is set, all objects found under the directory `base-location` are included.
+
+`include` and `exclude` are lists of regular expressions matched against the object's full URI. An
+object is included if it matches at least one `include` pattern (or `include` is omitted) and does
+not match any `exclude` pattern.
+
+**Example** โ include only JPEG and PNG images, but exclude thumbnails:
+
+```json
+{
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"],
+ "exclude": [".*thumbs/.*"]
+ }
+}
+```
+
+### Example: creating a directory
+
+```json
+POST /polaris/v1/{prefix}/namespaces/{namespace}/directories
+
+{
+ "name": "product-images",
+ "base-location": "s3://warehouse/product-images/",
+ "filter": {
+ "include": [".*\\.jpg$", ".*\\.png$"]
+ }
+}
+```
+
+## REST endpoints
+
+Directories are managed through the following REST endpoints. All paths are relative to the catalog
+base URL.
+
+### List directories
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Returns the list of directory identifiers in the given namespace.
+
+### Create a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories`
+
+Creates a new directory and its corresponding Iceberg table. The request body must contain the
+directory configuration (see [Configuration](#configuration)).
+
+### Get directory details
+
+**GET** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Returns the full configuration of the specified directory, including its filter.
+
+### Update a directory
+
+**POST** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Updates the `filter` and `properties` of the directory. Only the fields present in the request
+are changed, and a provided field replaces the existing value. The `name` and `base-location` cannot be changed.
+The directory table and its history are preserved, and the new filter applies from the next scan.
+
+### Drop a directory
+
+**DELETE** `/polaris/v1/{prefix}/namespaces/{namespace}/directories/{directory}`
+
+Removes the directory and its associated Iceberg table from the namespace.
+
+## Directory table
+
+When a directory is created, Polaris creates an Iceberg table in the same namespace using the directory
+`name` followed by `__inventory` as the table name. The table uses the following schema:
+
+| Field Id | Field Name | Type | Required | Description |
+|----------|------------|------|----------|-------------|
+| 1 | `file_uri` | `string` | Yes | The fully qualified URI of the object (for example `s3://my-bucket/images/photo.jpg`). |
+| 2 | `content_type` | `string` | No | The MIME content type (RFC 2045) of the object (for example `image/jpeg`, `application/pdf`). |
+| 3 | `size` | `long` | No | The size of the object in bytes. |
+| 4 | `checksum_algorithm` | `string` | No | The name of the checksum algorithm (for example `MD5`, `SHA-256`, `CRC32`). |
Review Comment:
I would keep it per row. Object stores don't guarantee one algorithm for every object under a prefix: on S3, each object can be uploaded with a different checksum (CRC32, SHA-256, ...), and `ListObjectsV2` reports the algorithm per object.
With a per-row algorithm, a scanner can record what the store already reports instead of reading every object to compute a single one (expensive for large videos for instance). It also keeps rows correct if the configured algorithm changes later: old rows keep the algorithm they were computed with until the next scan.
A directory-level preferred algorithm would fit in the `scanner-config` as a follow-up, and the column would still describe what was used. I clarified this in the column description.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org