You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@polaris.apache.org by yu...@apache.org on 2026/09/24 17:54:09 UTC
(polaris) branch main updated: Reject an unpaged listing that does not fit the configured maximum (#5553)
This is an automated email from the ASF dual-hosted git repository.
flyrain pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/polaris.git
The following commit(s) were added to refs/heads/main by this push:
new b900a3f6a9 Reject an unpaged listing that does not fit the configured maximum (#5553)
b900a3f6a9 is described below
commit b900a3f6a9660850e4e01cebc98ea8f50865424e
Author: Ayush Saxena <ay...@apache.org>
AuthorDate: Thu Sep 24 23:24:03 2026 +0530
Reject an unpaged listing that does not fit the configured maximum (#5553)
---
CHANGELOG.md | 15 ++--
.../semanticmodel/SemanticModelCatalogHandler.java | 23 +-----
.../polaris/core/config/FeatureConfiguration.java | 11 ++-
.../core/persistence/pagination/PageToken.java | 5 +-
.../service/catalog/common/CatalogHandler.java | 36 +++++++++
.../generic/GenericTableCatalogHandler.java | 16 +---
.../catalog/iceberg/IcebergCatalogHandler.java | 90 +++++++++++-----------
.../catalog/iceberg/IcebergCatalogAdapterTest.java | 44 +++++++----
.../catalog/iceberg/IcebergCatalogHandlerTest.java | 68 ++++++++++++++++
.../config-sections/flags-polaris_features.md | 2 +-
10 files changed, 201 insertions(+), 109 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f3b01808ab..797eb4df54 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -87,9 +87,10 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
- Internal JWTs minted before credentials-generation binding (tokens without the `polaris-cv` claim) can no longer be used as subject tokens in token exchange; they remain valid as bearer tokens until expiry. During a rolling upgrade, an old node may still mint claim-less tokens: exchanging such a token on any already-upgraded node fails with `invalid_grant`, so clients can see intermittent exchange failures until the last old node is gone; after that, rejection is consistent.
- `LIST_PAGINATION_ENABLED` now defaults to true. List APIs honor pagination parameters and reject
invalid values. Clients must follow next-page-token to retrieve all results when requesting a page
- size or when a positive LIST_PAGINATION_MAX_PAGE_SIZE limits local catalog listings. Otherwise,
- requests without pagination parameters still return all results. To keep the previous behavior,
- set `LIST_PAGINATION_ENABLED=false` or the catalog property `polaris.config.list-pagination-enabled=false`.
+ size or supplying a page token. A request that supplies neither still returns all results, unless
+ a positive `LIST_PAGINATION_MAX_PAGE_SIZE` is configured and the result does not fit, in which
+ case it is rejected rather than truncated. To keep the previous behavior, set
+ `LIST_PAGINATION_ENABLED=false` or the catalog property `polaris.config.list-pagination-enabled=false`.
- The `PolarisPrincipal` interface has evolved. The `getAttributes()` method now returns
`org.apache.polaris.core.collection.ImmutableAttributeMap`. The attribute keys were moved to a
new `org.apache.polaris.core.auth.PolarisPrincipalAttributes` class.
@@ -128,10 +129,10 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
For local catalogs the maximum takes effect only when `LIST_PAGINATION_ENABLED` is true, since
with pagination disabled the requested page size is ignored and the full result set is returned;
for federated catalogs it always applies, because Polaris paginates those listings itself.
- Setting a maximum deviates from the Iceberg REST specification, which requires a request that
- does not supply a `pageToken` to receive the complete result with a null `next-page-token`: such
- a request is then truncated to the maximum and answered with a continuation token, so a client
- that does not follow continuations sees only the first page.
+ A request that supplies neither `pageToken` nor `pageSize` asks for the complete listing, so when
+ the result does not fit the maximum it is rejected rather than truncated and answered with a
+ continuation token. An empty `pageToken` starts a paginated listing and is capped like any other
+ paginated request. This applies to the Iceberg, generic-table and semantic-model listings alike.
- Table commits whose base metadata is already stale now fail before the new metadata file is
written, saving an object-storage write and delete per conflict and returning the `409` to the
client sooner.
diff --git a/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogHandler.java b/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogHandler.java
index 12855df070..b9e8bb50ab 100644
--- a/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogHandler.java
+++ b/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogHandler.java
@@ -18,8 +18,6 @@
*/
package org.apache.polaris.service.catalog.semanticmodel;
-import static org.apache.polaris.core.config.FeatureConfiguration.LIST_PAGINATION_MAX_PAGE_SIZE;
-
import java.util.List;
import org.apache.iceberg.catalog.Namespace;
import org.apache.polaris.core.auth.AuthorizationRequest;
@@ -27,8 +25,6 @@ import org.apache.polaris.core.auth.AuthorizationState;
import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
import org.apache.polaris.core.auth.SingleTargetAuthorizationIntent;
import org.apache.polaris.core.catalog.PolarisCatalogHelpers;
-import org.apache.polaris.core.config.FeatureConfiguration;
-import org.apache.polaris.core.entity.CatalogEntity;
import org.apache.polaris.core.entity.PolarisEntityType;
import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
import org.apache.polaris.core.persistence.pagination.PageToken;
@@ -88,7 +84,10 @@ public abstract class SemanticModelCatalogHandler extends CatalogHandler {
PageToken pageRequest =
PageToken.build(pageToken, pageSize, maxPageSize(), this::shouldDecodeToken);
- return semanticModelCatalog.listSemanticModels(namespace, pageRequest);
+ ListSemanticModelsResponse response =
+ semanticModelCatalog.listSemanticModels(namespace, pageRequest);
+ rejectIncompleteListing(pageToken, pageSize, response.getNextPageToken());
+ return response;
}
public LoadSemanticModelResponse loadSemanticModel(SemanticModelIdentifier identifier) {
@@ -111,20 +110,6 @@ public abstract class SemanticModelCatalogHandler extends CatalogHandler {
semanticModelCatalog.dropSemanticModel(identifier);
}
- private int maxPageSize() {
- CatalogEntity catalogEntity = resolutionManifest.getResolvedCatalogEntity();
- return catalogEntity == null
- ? realmConfig().getConfig(LIST_PAGINATION_MAX_PAGE_SIZE)
- : realmConfig().getConfig(LIST_PAGINATION_MAX_PAGE_SIZE, catalogEntity);
- }
-
- private boolean shouldDecodeToken() {
- CatalogEntity catalogEntity = resolutionManifest.getResolvedCatalogEntity();
- return catalogEntity == null
- ? realmConfig().getConfig(FeatureConfiguration.LIST_PAGINATION_ENABLED)
- : realmConfig().getConfig(FeatureConfiguration.LIST_PAGINATION_ENABLED, catalogEntity);
- }
-
private void authorizeBasicSemanticModelOperationOrThrow(
PolarisAuthorizableOperation op, SemanticModelIdentifier identifier) {
Namespace namespace = Namespace.of(identifier.getNamespace().toArray(new String[0]));
diff --git a/polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java b/polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java
index 2767f29b3c..7c527a068f 100644
--- a/polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java
+++ b/polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java
@@ -495,12 +495,11 @@ public class FeatureConfiguration<T> extends PolarisConfiguration<T> {
+ " when LIST_PAGINATION_ENABLED is true; with pagination disabled the requested page"
+ " size is ignored and the full result set is returned. For federated catalogs it"
+ " always applies, because Polaris paginates those listings itself. Values <= 0 mean"
- + " unlimited (no max page size enforced), which is the default. Setting a maximum"
- + " deviates from the Iceberg REST specification, which requires a request that"
- + " does not supply a pageToken to receive the complete result with a null"
- + " next-page-token: such a request is then truncated to the maximum and answered"
- + " with a continuation token, so a client that does not follow continuations sees"
- + " only the first page.")
+ + " unlimited (no max page size enforced), which is the default. A request that"
+ + " supplies neither pageToken nor pageSize asks for the complete listing, so when"
+ + " the result does not fit the maximum it is rejected rather than truncated and"
+ + " answered with a continuation token. An empty pageToken starts a paginated"
+ + " listing and is capped like any other paginated request.")
.defaultValue(-1)
.buildFeatureConfiguration();
diff --git a/polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java b/polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java
index 69c81499f4..99a7019750 100644
--- a/polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java
+++ b/polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java
@@ -87,8 +87,9 @@ public interface PageToken {
* @param requestedPageSize optional page size for the next page. If not set, the page size of the
* previous page (encoded in the page token string) will be reused.
* @param maxPageSize the largest page size to honour, or zero or less for unlimited. When set, a
- * request that asks for no page size is paginated at this size rather than reading
- * everything.
+ * request that asks for no page size is bounded at this size, so the resulting page carries a
+ * continuation token when the result would have overflowed. What the caller does with that is
+ * its own concern.
* @see Page#encodedResponseToken()
*/
static PageToken build(
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/catalog/common/CatalogHandler.java b/runtime/service/src/main/java/org/apache/polaris/service/catalog/common/CatalogHandler.java
index ddae7a870f..d4fe7dfe9f 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/catalog/common/CatalogHandler.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/catalog/common/CatalogHandler.java
@@ -30,6 +30,7 @@ import java.util.List;
import org.apache.iceberg.catalog.Namespace;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.exceptions.BadRequestException;
import org.apache.polaris.core.auth.AuthorizationIntent;
import org.apache.polaris.core.auth.AuthorizationRequest;
import org.apache.polaris.core.auth.AuthorizationState;
@@ -39,9 +40,11 @@ import org.apache.polaris.core.auth.PolarisPrincipal;
import org.apache.polaris.core.auth.RenameAuthorizationIntent;
import org.apache.polaris.core.auth.SingleTargetAuthorizationIntent;
import org.apache.polaris.core.catalog.PolarisCatalogHelpers;
+import org.apache.polaris.core.config.FeatureConfiguration;
import org.apache.polaris.core.config.RealmConfig;
import org.apache.polaris.core.context.CallContext;
import org.apache.polaris.core.context.RealmContext;
+import org.apache.polaris.core.entity.CatalogEntity;
import org.apache.polaris.core.entity.PolarisEntitySubType;
import org.apache.polaris.core.entity.PolarisEntityType;
import org.apache.polaris.core.persistence.PolarisMetaStoreManager;
@@ -53,6 +56,7 @@ import org.apache.polaris.core.persistence.resolver.ResolverPath;
import org.apache.polaris.core.persistence.resolver.ResolverStatus;
import org.apache.polaris.service.types.PolicyIdentifier;
import org.immutables.value.Value;
+import org.jspecify.annotations.Nullable;
/**
* An ABC for catalog wrappers which provides authorize methods that should be called before a
@@ -91,6 +95,38 @@ public abstract class CatalogHandler {
@SuppressWarnings("immutables:incompat")
protected PolarisResolutionManifest resolutionManifest = null;
+ protected boolean shouldDecodeToken() {
+ CatalogEntity catalogEntity = resolutionManifest.getResolvedCatalogEntity();
+ return catalogEntity == null
+ ? realmConfig().getConfig(FeatureConfiguration.LIST_PAGINATION_ENABLED)
+ : realmConfig().getConfig(FeatureConfiguration.LIST_PAGINATION_ENABLED, catalogEntity);
+ }
+
+ /** The configured page size ceiling; zero or less means unlimited. */
+ protected int maxPageSize() {
+ CatalogEntity catalogEntity = resolutionManifest.getResolvedCatalogEntity();
+ return catalogEntity == null
+ ? realmConfig().getConfig(FeatureConfiguration.LIST_PAGINATION_MAX_PAGE_SIZE)
+ : realmConfig()
+ .getConfig(FeatureConfiguration.LIST_PAGINATION_MAX_PAGE_SIZE, catalogEntity);
+ }
+
+ /**
+ * A request that supplies neither {@code pageToken} nor {@code pageSize} asks for the complete
+ * listing, which the Iceberg REST specification answers with every result and a null
+ * next-page-token. When a configured maximum prevents that, fail rather than return a page that
+ * looks complete and is not. An empty {@code pageToken} starts a paginated listing, so it is
+ * capped like any other paginated request.
+ */
+ protected void rejectIncompleteListing(
+ @Nullable String pageToken, @Nullable Integer pageSize, @Nullable String nextPageToken) {
+ if (pageToken == null && pageSize == null && nextPageToken != null) {
+ throw new BadRequestException(
+ "Listing does not fit the maximum page size of %s; supply a pageToken to paginate",
+ maxPageSize());
+ }
+ }
+
/** Initialize the catalog once authorized. Called after all `authorize...` methods. */
protected abstract void initializeCatalog();
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/catalog/generic/GenericTableCatalogHandler.java b/runtime/service/src/main/java/org/apache/polaris/service/catalog/generic/GenericTableCatalogHandler.java
index 6cec0e9b94..e8c21a1eae 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/catalog/generic/GenericTableCatalogHandler.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/catalog/generic/GenericTableCatalogHandler.java
@@ -107,6 +107,7 @@ public abstract class GenericTableCatalogHandler extends CatalogHandler {
PageToken pageRequest =
PageToken.build(pageToken, pageSize, maxPageSize(), this::shouldDecodeToken);
Page<TableIdentifier> page = genericTableCatalog.listGenericTables(parent, pageRequest);
+ rejectIncompleteListing(pageToken, pageSize, page.encodedResponseToken());
return ListGenericTablesResponse.builder()
.setIdentifiers(new LinkedHashSet<>(page.items()))
.setNextPageToken(page.encodedResponseToken())
@@ -162,19 +163,4 @@ public abstract class GenericTableCatalogHandler extends CatalogHandler {
return LoadGenericTableResponse.builder().setTable(loadedTable).build();
}
-
- private boolean shouldDecodeToken() {
- CatalogEntity catalogEntity = resolutionManifest.getResolvedCatalogEntity();
- return catalogEntity == null
- ? realmConfig().getConfig(FeatureConfiguration.LIST_PAGINATION_ENABLED)
- : realmConfig().getConfig(FeatureConfiguration.LIST_PAGINATION_ENABLED, catalogEntity);
- }
-
- private int maxPageSize() {
- CatalogEntity catalogEntity = resolutionManifest.getResolvedCatalogEntity();
- return catalogEntity == null
- ? realmConfig().getConfig(FeatureConfiguration.LIST_PAGINATION_MAX_PAGE_SIZE)
- : realmConfig()
- .getConfig(FeatureConfiguration.LIST_PAGINATION_MAX_PAGE_SIZE, catalogEntity);
- }
}
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java b/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java
index 273ddc9510..ca02389683 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java
@@ -20,8 +20,6 @@ package org.apache.polaris.service.catalog.iceberg;
import static java.util.Objects.requireNonNull;
import static org.apache.polaris.core.config.FeatureConfiguration.ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING;
-import static org.apache.polaris.core.config.FeatureConfiguration.LIST_PAGINATION_ENABLED;
-import static org.apache.polaris.core.config.FeatureConfiguration.LIST_PAGINATION_MAX_PAGE_SIZE;
import static org.apache.polaris.service.catalog.AccessDelegationMode.VENDED_CREDENTIALS;
import static org.apache.polaris.service.catalog.common.ExceptionUtils.alreadyExistsExceptionForTableLikeEntity;
import static org.apache.polaris.service.catalog.common.ExceptionUtils.noSuchNamespaceException;
@@ -219,39 +217,6 @@ public abstract class IcebergCatalogHandler extends CatalogHandler implements Au
return catalogEntity;
}
- private boolean shouldDecodeToken() {
- return realmConfig().getConfig(LIST_PAGINATION_ENABLED, getResolvedCatalogEntity());
- }
-
- /** The configured page size ceiling; zero or less means unlimited. */
- private int maxPageSize() {
- return realmConfig().getConfig(LIST_PAGINATION_MAX_PAGE_SIZE, getResolvedCatalogEntity());
- }
-
- /**
- * Bounds the page size forwarded to a federated catalog, using the same rule as {@link
- * PageTokenUtil#boundPageSize}. An absent page size becomes the maximum, so the remote result set
- * is not returned whole.
- */
- private @Nullable Integer boundedPageSize(@Nullable Integer requestedPageSize) {
- OptionalInt requested =
- requestedPageSize == null ? OptionalInt.empty() : OptionalInt.of(requestedPageSize);
- OptionalInt bounded = PageTokenUtil.boundPageSize(requested, maxPageSize());
- return bounded.isPresent() ? bounded.getAsInt() : null;
- }
-
- /**
- * The page token forwarded to a federated catalog. {@code CatalogHandlerUtils} returns the whole
- * result set when no token is given, so when a maximum is configured the specification's initial
- * page token is supplied instead, making the listing paginate from the first request as it does
- * for a local catalog.
- */
- private @Nullable String boundedPageToken(@Nullable String pageToken) {
- return pageToken == null && maxPageSize() > 0
- ? CatalogHandlerUtils.INITIAL_PAGE_TOKEN
- : pageToken;
- }
-
@Override
protected void initializeCatalog() {
CatalogEntity resolvedCatalogEntity = getResolvedCatalogEntity();
@@ -301,19 +266,48 @@ public abstract class IcebergCatalogHandler extends CatalogHandler implements Au
this.viewCatalog = (baseCatalog instanceof ViewCatalog) ? (ViewCatalog) baseCatalog : null;
}
+ /**
+ * Bounds the page size forwarded to a federated catalog, using the same rule as {@link
+ * PageTokenUtil#boundPageSize}. An absent page size becomes the maximum, so the remote result set
+ * is not returned whole.
+ */
+ private @Nullable Integer boundedPageSize(@Nullable Integer requestedPageSize) {
+ OptionalInt requested =
+ requestedPageSize == null ? OptionalInt.empty() : OptionalInt.of(requestedPageSize);
+ OptionalInt bounded = PageTokenUtil.boundPageSize(requested, maxPageSize());
+ return bounded.isPresent() ? bounded.getAsInt() : null;
+ }
+
+ /**
+ * The page token forwarded to a federated catalog. {@code CatalogHandlerUtils} returns the whole
+ * result set when no token is given, so when a maximum is configured the specification's initial
+ * page token is supplied instead. That bounds the response and surfaces a continuation token when
+ * the result would have overflowed, which {@link #rejectIncompleteListing} then turns into an
+ * error for a request that asked for the complete listing.
+ */
+ private @Nullable String boundedPageToken(@Nullable String pageToken) {
+ return pageToken == null && maxPageSize() > 0
+ ? CatalogHandlerUtils.INITIAL_PAGE_TOKEN
+ : pageToken;
+ }
+
public ListNamespacesResponse listNamespaces(
Namespace parent, String pageToken, Integer pageSize) {
PolarisAuthorizableOperation op = PolarisAuthorizableOperation.LIST_NAMESPACES;
authorizeBasicNamespaceOperationOrThrow(op, parent);
if (isFederated) {
- return catalogHandlerUtils()
- .listNamespaces(
- namespaceCatalog, parent, boundedPageToken(pageToken), boundedPageSize(pageSize));
+ ListNamespacesResponse response =
+ catalogHandlerUtils()
+ .listNamespaces(
+ namespaceCatalog, parent, boundedPageToken(pageToken), boundedPageSize(pageSize));
+ rejectIncompleteListing(pageToken, pageSize, response.nextPageToken());
+ return response;
} else {
PageToken pageRequest =
PageToken.build(pageToken, pageSize, maxPageSize(), this::shouldDecodeToken);
var results = ((LocalIcebergCatalog) baseCatalog).listNamespaces(parent, pageRequest);
+ rejectIncompleteListing(pageToken, pageSize, results.encodedResponseToken());
return ListNamespacesResponse.builder()
.addAll(results.items())
.nextPageToken(results.encodedResponseToken())
@@ -407,13 +401,17 @@ public abstract class IcebergCatalogHandler extends CatalogHandler implements Au
authorizeBasicNamespaceOperationOrThrow(op, namespace);
if (isFederated) {
- return catalogHandlerUtils()
- .listTables(
- baseCatalog, namespace, boundedPageToken(pageToken), boundedPageSize(pageSize));
+ ListTablesResponse response =
+ catalogHandlerUtils()
+ .listTables(
+ baseCatalog, namespace, boundedPageToken(pageToken), boundedPageSize(pageSize));
+ rejectIncompleteListing(pageToken, pageSize, response.nextPageToken());
+ return response;
} else {
PageToken pageRequest =
PageToken.build(pageToken, pageSize, maxPageSize(), this::shouldDecodeToken);
var results = ((LocalIcebergCatalog) baseCatalog).listTables(namespace, pageRequest);
+ rejectIncompleteListing(pageToken, pageSize, results.encodedResponseToken());
return ListTablesResponse.builder()
.addAll(results.items())
.nextPageToken(results.encodedResponseToken())
@@ -1644,9 +1642,12 @@ public abstract class IcebergCatalogHandler extends CatalogHandler implements Au
if (isFederated) {
if (baseCatalog instanceof ViewCatalog viewCatalog) {
- return catalogHandlerUtils()
- .listViews(
- viewCatalog, namespace, boundedPageToken(pageToken), boundedPageSize(pageSize));
+ ListTablesResponse response =
+ catalogHandlerUtils()
+ .listViews(
+ viewCatalog, namespace, boundedPageToken(pageToken), boundedPageSize(pageSize));
+ rejectIncompleteListing(pageToken, pageSize, response.nextPageToken());
+ return response;
}
throw new BadRequestException(
"Unsupported operation: listViews with baseCatalog type: %s",
@@ -1655,6 +1656,7 @@ public abstract class IcebergCatalogHandler extends CatalogHandler implements Au
PageToken pageRequest =
PageToken.build(pageToken, pageSize, maxPageSize(), this::shouldDecodeToken);
var results = ((LocalIcebergCatalog) baseCatalog).listViews(namespace, pageRequest);
+ rejectIncompleteListing(pageToken, pageSize, results.encodedResponseToken());
return ListTablesResponse.builder()
.addAll(results.items())
.nextPageToken(results.encodedResponseToken())
diff --git a/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapterTest.java b/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapterTest.java
index eb63a6c6e2..b780d05430 100644
--- a/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapterTest.java
+++ b/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapterTest.java
@@ -31,6 +31,7 @@ import java.util.stream.Stream;
import org.apache.iceberg.Schema;
import org.apache.iceberg.catalog.Namespace;
import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.BadRequestException;
import org.apache.iceberg.inmemory.InMemoryCatalog;
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
@@ -371,48 +372,61 @@ public class IcebergCatalogAdapterTest {
}
/**
- * A federated listing with no page token at all must still be bounded: the whole remote result
- * set would otherwise be returned in one response.
+ * A federated listing that supplies neither a page token nor a page size asks for the complete
+ * result, so it is rejected when that does not fit the configured maximum rather than served a
+ * page that looks complete. A listing that does fit is answered in full.
*/
@Test
- void testFederatedListingsWithoutAPageTokenAreBounded() throws IOException {
+ void testFederatedListingsWithoutAPageTokenAreRejectedWhenTheyDoNotFit() throws IOException {
try (InMemoryCatalog inMemoryCatalog = new InMemoryCatalog()) {
inMemoryCatalog.initialize("inMemory", Map.of());
mockCatalogAdapter(inMemoryCatalog);
+ // One more entity than the LIST_PAGINATION_MAX_PAGE_SIZE configured above
int entityCount = 101;
for (int i = 0; i < entityCount; ++i) {
inMemoryCatalog.createNamespace(Namespace.of("ns" + i));
inMemoryCatalog.createTable(TableIdentifier.of("ns0", "table" + i), new Schema());
}
- ListNamespacesResponse namespaces =
- (ListNamespacesResponse)
- catalogAdapter
- .listNamespaces(
+ Assertions.assertThatThrownBy(
+ () ->
+ catalogAdapter.listNamespaces(
FEDERATED_CATALOG_NAME,
null,
null,
null,
testServices.realmContext(),
- testServices.securityContext())
- .getEntity();
- Assertions.assertThat(namespaces.namespaces()).hasSize(100);
- Assertions.assertThat(namespaces.nextPageToken()).isNotNull();
+ testServices.securityContext()))
+ .isInstanceOf(BadRequestException.class)
+ .hasMessageContaining("100");
- ListTablesResponse tables =
+ Assertions.assertThatThrownBy(
+ () ->
+ catalogAdapter.listTables(
+ FEDERATED_CATALOG_NAME,
+ "ns0",
+ null,
+ null,
+ testServices.realmContext(),
+ testServices.securityContext()))
+ .isInstanceOf(BadRequestException.class)
+ .hasMessageContaining("100");
+
+ // ns1 holds no tables, so the same unpaged request fits and is answered in full.
+ ListTablesResponse empty =
(ListTablesResponse)
catalogAdapter
.listTables(
FEDERATED_CATALOG_NAME,
- "ns0",
+ "ns1",
null,
null,
testServices.realmContext(),
testServices.securityContext())
.getEntity();
- Assertions.assertThat(tables.identifiers()).hasSize(100);
- Assertions.assertThat(tables.nextPageToken()).isNotNull();
+ Assertions.assertThat(empty.identifiers()).isEmpty();
+ Assertions.assertThat(empty.nextPageToken()).isNull();
}
}
diff --git a/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerTest.java b/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerTest.java
index 24556ef758..80609b8c22 100644
--- a/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerTest.java
+++ b/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerTest.java
@@ -55,6 +55,7 @@ import org.apache.iceberg.catalog.Namespace;
import org.apache.iceberg.catalog.SupportsNamespaces;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.exceptions.BadRequestException;
import org.apache.iceberg.exceptions.NoSuchNamespaceException;
import org.apache.iceberg.exceptions.NoSuchTableException;
import org.apache.iceberg.exceptions.NoSuchViewException;
@@ -866,6 +867,73 @@ class IcebergCatalogHandlerTest {
verify(catalogHandlerUtils, never()).loadNamespace(any(), any());
}
+ /**
+ * A request that supplies neither a page token nor a page size asks for the complete listing, so
+ * a result that does not fit the configured maximum is an error rather than a page that looks
+ * complete and is not.
+ */
+ @Test
+ void listTablesRejectsAnUnpagedRequestThatDoesNotFit() {
+ LocalIcebergCatalog catalog = mock(LocalIcebergCatalog.class);
+ when(localCatalogFactory.createCatalog(any())).thenReturn(catalog);
+ when(catalog.listTables(eq(NS1), any(PageToken.class)))
+ .thenReturn(
+ Page.page(
+ PageToken.fromLimit(100),
+ new ArrayList<>(List.of(TABLE2)),
+ EntityIdToken.fromEntityId(1L)));
+ when(realmConfig.getConfig(LIST_PAGINATION_ENABLED, catalogEntity)).thenReturn(true);
+ when(realmConfig.getConfig(LIST_PAGINATION_MAX_PAGE_SIZE, catalogEntity)).thenReturn(100);
+
+ @SuppressWarnings("resource")
+ IcebergCatalogHandler handler = newHandler();
+ assertThatThrownBy(() -> handler.listTables(NS1, null, null))
+ .isInstanceOf(BadRequestException.class)
+ .hasMessageContaining("100");
+ }
+
+ /** The same request is answered in full, with no continuation token, when the result fits. */
+ @Test
+ void listTablesReturnsTheCompleteListWhenAnUnpagedRequestFits() {
+ LocalIcebergCatalog catalog = mock(LocalIcebergCatalog.class);
+ when(localCatalogFactory.createCatalog(any())).thenReturn(catalog);
+ when(catalog.listTables(eq(NS1), any(PageToken.class)))
+ .thenReturn(Page.fromItems(new ArrayList<>(List.of(TABLE2))));
+ when(realmConfig.getConfig(LIST_PAGINATION_ENABLED, catalogEntity)).thenReturn(true);
+ when(realmConfig.getConfig(LIST_PAGINATION_MAX_PAGE_SIZE, catalogEntity)).thenReturn(100);
+
+ @SuppressWarnings("resource")
+ IcebergCatalogHandler handler = newHandler();
+ var response = handler.listTables(NS1, null, null);
+
+ assertThat(response.identifiers()).containsExactly(TABLE2);
+ assertThat(response.nextPageToken()).isNull();
+ }
+
+ /**
+ * An empty page token is how a client starts a paginated listing, so it is capped like any other
+ * paginated request rather than rejected.
+ */
+ @Test
+ void listTablesPaginatesAnEmptyPageTokenRatherThanRejectingIt() {
+ LocalIcebergCatalog catalog = mock(LocalIcebergCatalog.class);
+ when(localCatalogFactory.createCatalog(any())).thenReturn(catalog);
+ when(catalog.listTables(eq(NS1), any(PageToken.class)))
+ .thenReturn(
+ Page.page(
+ PageToken.fromLimit(100),
+ new ArrayList<>(List.of(TABLE2)),
+ EntityIdToken.fromEntityId(1L)));
+ when(realmConfig.getConfig(LIST_PAGINATION_ENABLED, catalogEntity)).thenReturn(true);
+ when(realmConfig.getConfig(LIST_PAGINATION_MAX_PAGE_SIZE, catalogEntity)).thenReturn(100);
+
+ @SuppressWarnings("resource")
+ IcebergCatalogHandler handler = newHandler();
+ var response = handler.listTables(NS1, "", null);
+
+ assertThat(response.nextPageToken()).isNotNull();
+ }
+
/**
* A page token carries the page size it was minted with, so a token issued before the limit was
* configured (or a hand-crafted one) must not be able to escape the bound.
diff --git a/site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md b/site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md
index 265b4c2a10..741d062112 100644
--- a/site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md
+++ b/site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md
@@ -460,7 +460,7 @@ If set to true, pagination for APIs like listTables is enabled.
##### `polaris.features."LIST_PAGINATION_MAX_PAGE_SIZE"`
-The largest page size a client may request for APIs like listTables. Larger requested page sizes are reduced to this value. The Iceberg REST specification treats the requested page size as an upper bound. For local catalogs this takes effect only when LIST_PAGINATION_ENABLED is true; with pagination disabled the requested page size is ignored and the full result set is returned. For federated catalogs it always applies, because Polaris paginates those listings itself. Values <= 0 mean un [...]
+The largest page size a client may request for APIs like listTables. Larger requested page sizes are reduced to this value. The Iceberg REST specification treats the requested page size as an upper bound. For local catalogs this takes effect only when LIST_PAGINATION_ENABLED is true; with pagination disabled the requested page size is ignored and the full result set is returned. For federated catalogs it always applies, because Polaris paginates those listings itself. Values <= 0 mean un [...]
- **Type:** `Integer`
- **Default:** `-1`