You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "eric-maynard (via GitHub)" <gi...@apache.org> on 2025/05/05 17:19:32 UTC
[PR] Interface changes for pagination [polaris]
eric-maynard opened a new pull request, #1528:
URL: https://github.com/apache/polaris/pull/1528
The Apache Iceberg REST catalog spec [describes](https://github.com/apache/iceberg/blob/f7c6d57a03282516c6f988fa23a809416cd765b1/open-api/rest-catalog-open-api.yaml#L258) how APIs like `listNamespaces` can implement pagination, but Polaris currently does not implement pagination.
This PR introduces a framework that will allow us to enable pagination for these APIs and that can also be extended to other APIs like `listCatalogs`.
<hr>
This is a modified version of #273 that removes the actual pagination implementation at the persistence level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard merged PR #1528:
URL: https://github.com/apache/polaris/pull/1528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080152035
##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -169,6 +170,23 @@ public static boolean isCreate(UpdateTableRequest request) {
return isCreate;
}
+ public ListNamespacesResponse listNamespaces(
+ Namespace parent, String pageToken, Integer pageSize) {
+ PolarisAuthorizableOperation op = PolarisAuthorizableOperation.LIST_NAMESPACES;
+ authorizeBasicNamespaceOperationOrThrow(op, parent);
+
+ if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
+ Page<Namespace> results = polarisCatalog.listNamespaces(parent, pageToken, pageSize);
+ return ListNamespacesResponse.builder()
+ .addAll(results.items)
+ .nextPageToken(results.pageToken.toString())
+ .build();
+ } else {
+ return CatalogHandlers.listNamespaces(
+ namespaceCatalog, parent, pageToken, String.valueOf(pageSize));
Review Comment:
ah, you're right. In most cases we should stop using `CatalogHandlers` methods as a fallback during federation or at least add our own logic, I think. I think we can do this as a followup
##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -169,6 +170,23 @@ public static boolean isCreate(UpdateTableRequest request) {
return isCreate;
}
+ public ListNamespacesResponse listNamespaces(
+ Namespace parent, String pageToken, Integer pageSize) {
+ PolarisAuthorizableOperation op = PolarisAuthorizableOperation.LIST_NAMESPACES;
+ authorizeBasicNamespaceOperationOrThrow(op, parent);
+
+ if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
+ Page<Namespace> results = polarisCatalog.listNamespaces(parent, pageToken, pageSize);
+ return ListNamespacesResponse.builder()
+ .addAll(results.items)
+ .nextPageToken(results.pageToken.toString())
+ .build();
+ } else {
+ return CatalogHandlers.listNamespaces(
+ namespaceCatalog, parent, pageToken, String.valueOf(pageSize));
Review Comment:
ah, you're right. In most cases we should stop using `CatalogHandlers` methods as a fallback during federation or at least add our own logic, I think. Probably we can do this as a followup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078370027
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
Your comment gets at a very real issue that I was grappling with when I wrote this... maybe we can come up with a better approach.
Basically, each PageToken has a toString method, and a static fromString method. I would ideally like to enforce via the ABC that each PageToken implementation overrides these, but in Java you can't override a static method like that for some reason.
Also, we want to enforce that toString and fromString always round-trip cleanly, and I want to encapsulate all the PageToken-specific logic in the child classes rather than putting everything into PageToken.fromString.
So to this end, I came up with this toComponents / fromComponents / expectedComponents pattern where toComponents and fromComponents are expected to work with exactly expectedComponents elements. In the case of `ReadEverythingPageToken` expectedComponents is 0.
If there is a better way to implement this kind of polymorphic serde for PageToken I am very open to trying it, thanks for taking a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2073950129
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
Review Comment:
Reviving the discussion on the name below...
It's currently called `PageToken` because it encapsulates the data in `page-token` sent by the client. It also encapsulates `page-size` and so there has been some objection to the naming.
At its core, this `PageToken` is used to describe how persistence should build a page of data. It is, by itself, not a sufficient "page request" in that it doesn't contain information like the type of entity to search for. But it is used in conjunction to with a request to filter results down.
Because the Iceberg spec is vague about the `page-token` implementation -- calling it "opaque" -- I feel that a page token which contains information about the size of the page is still consistent with the spec. `page-size` is still respected when it's present in a request alongside `page-token`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078481034
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
From a more general perspective, I believe the server cannot be expected to always comply with the client's page size.
1) To safeguard against DOS attacks
2) For technical reasons:
* A persistence query may return less data than expected. It might be preferable to finish the request with a smaller page and let the client re-request, than keep polling the backend.
* A communication protocol may have restrictions based on the message size, not the number of items encoded in the message.
All in all, I think clients should not expect responses to have exactly as many entries as requested even when the result set is not exhausted. Polaris may be able to satisfy that expectation in most cases, but we should not make it a guaranteed feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078881363
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
For at least the first iteration, I want to take a hard line on complying with the spec. That means that if there's no page token, we return everything.
Today, we just return all results no matter what the user asks for. I think it's okay to keep returning all results if the user asks for it, but at least now we will give users the option to ask for something else. If nothing else it's an incremental improvement.
wrt. null vs empty string, I'm a bit paranoid about us accidentally paginating results when the user doesn't expect it. So I think it's better to err on the side of the current (and still spec-compliant) behavior of just returning everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078564896
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
but now I did notice that the REST API [spec](https://github.com/apache/iceberg/blob/f7c6d57a03282516c6f988fa23a809416cd765b1/open-api/rest-catalog-open-api.yaml#L1689-L1691) indicates that the presence of an empty `PageToken` should be interpreted as the client's intention to use pagination.
That statement in itself is fine. Given my comment above, I think it might be worth interpreting `null` and empty page tokens differently. I'd still suggest to use a possibly large, but limited page size for the `null` token. We could use a smaller default page size for the empty token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2079930782
##########
extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java:
##########
@@ -450,27 +455,33 @@ public List<EntityNameLookupRecord> lookupEntityActiveBatchInCurrentTxn(
entity.getParentId(),
entity.getName(),
entity.getTypeCode(),
- entity.getSubTypeCode()));
+ entity.getSubTypeCode()),
+ pageToken);
}
@Override
- public @Nonnull <T> List<T> listEntitiesInCurrentTxn(
+ public @Nonnull <T> Page<T> listEntitiesInCurrentTxn(
@Nonnull PolarisCallContext callCtx,
long catalogId,
long parentId,
@Nonnull PolarisEntityType entityType,
- int limit,
@Nonnull Predicate<PolarisBaseEntity> entityFilter,
- @Nonnull Function<PolarisBaseEntity, T> transformer) {
+ @Nonnull Function<PolarisBaseEntity, T> transformer,
+ @Nonnull PageToken pageToken) {
// full range scan under the parent for that type
- return this.store
- .lookupFullEntitiesActive(localSession.get(), catalogId, parentId, entityType)
- .stream()
- .map(ModelEntity::toEntity)
- .filter(entityFilter)
- .limit(limit)
- .map(transformer)
- .collect(Collectors.toList());
+ Stream<PolarisBaseEntity> data =
+ this.store
+ .lookupFullEntitiesActive(
+ localSession.get(), catalogId, parentId, entityType, pageToken)
+ .stream()
+ .map(ModelEntity::toEntity)
+ .filter(entityFilter);
+
+ if (pageToken instanceof HasPageSize hasPageSize) {
Review Comment:
I'm not sure this is functionally correct. If the token is a sub-class of `HasPageSize` but has other information that the client provided, but EclipseLink does not recognise, I do not think it would be correct to just use the page size from the token.
I believe we should check for a specific type here and fail on all types, whose details cannot be fully processed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078469304
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
Review Comment:
I'm ok with manual builders but I made about comment about "api skew". Closing this thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078361797
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java:
##########
@@ -408,4 +414,7 @@ boolean hasChildren(
default BasePersistence detach() {
return this;
}
+
+ @Nonnull
+ PageToken.PageTokenBuilder<?> pageTokenBuilder();
Review Comment:
The intent here is that different metastores could have different PageToken implementations that they need to use. For example, it seems like EclipseLink may just not support pagination, which is fine.
Practically, we need a way for persistence to be able to say "Ok, I fetched the first N results, here's a page token that will give you the next page". The builder which constructs that page token varies by persistence implementation. But if a different code structure would work better here I'm very open to it. Maybe we need something more like a factory? I wanted to avoid creating to many unnecessary types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078881363
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
For at least the first iteration, I want to take a hard line on complying with the spec. That means that if there's no page token, we return everything.
Today, we just return all results no matter what the user asks for. I think it's okay to keep returning all results if the user asks for it, but at least now we will give users the option to ask for something else. If nothing else it's an incremental improvement.
wrt. null vs empty string, I'm a bit paranoid about us accidentally paginating results when the user doesn't expect it. So there too I think it's better to err on the side of the current (and still spec-compliant) behavior of just returning everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080131391
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadFromStartPageToken.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation that has a page size, but no start offset. This can be used to
+ * represent a `limit`. When updated, it returns a token with the same semantics.
+ */
+public class ReadFromStartPageToken extends PageToken implements HasPageSize {
+
+ public static final String PREFIX = "read-from-start";
+
+ private final int pageSize;
+
+ public ReadFromStartPageToken(int pageSize) {
+ this.pageSize = pageSize;
+ }
+
+ @Override
+ public int getPageSize() {
+ return pageSize;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("%s/%d", PREFIX, pageSize);
+ }
+
+ @Override
+ protected PageToken updated(List<?> newData) {
+ return new ReadFromStartPageToken(pageSize);
Review Comment:
This should never be used for pagination, it just represents a limit. Let me change this behavior and make this clear in the javadoc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078481034
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
From a more general perspective, I believe the server cannot be expected to always comply with the client's page size.
1) To safeguard against DOS attacks
2) For technical reasons:
* A persistence query may return less data than expected. It might be preferable to finish the request with a smaller page and let the client re-request, than keep polling the backend.
* A communication protocol may have restrictions based on the message size, not the number of items encoded in the message.
All in all, I think clients should not expect responses to have exactly as many entries as requested even when the result set is not exhausted. Polaris may be able to satisfy that expected in most cases, but we should not make it a guaranteed feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078509490
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
This is mostly governed by the Iceberg spec and our goal should be to comply with that.
What you wrote is true, that the server may return more or less records than the requested page size (whether embedded in the token or explicitly requested by the client). But we still need a way for the client to request a given page size as per the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078881363
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
For at least the first iteration, I want to take a hard line on complying with the spec. Today, we just return all results no matter what the user asks for. I think it's okay to keep returning all results if the user asks for it, but at least now we will give users the option to ask for something else.
wrt. null vs empty string, I'm a bit paranoid about us accidentally paginating results when the user doesn't expect it. So I think it's better to err on the side of the current (and still spec-compliant) behavior of just returning everything).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2079934994
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ /** Build a new PageToken that reads everything */
+ public static PageToken readEverything() {
+ return build(null, null);
+ }
+
+ /** Build a new PageToken from an input String, without a specified page size */
+ public static PageToken fromString(String token) {
+ return build(token, null);
+ }
+
+ /** Build a new PageToken from a limit */
+ public static PageToken fromLimit(Integer pageSize) {
+ return build(null, pageSize);
+ }
+
+ /** Build a {@link PageToken} from the input string and page size */
+ public static PageToken build(String token, Integer pageSize) {
+ if (token == null || token.isEmpty()) {
+ if (pageSize != null) {
+ return new ReadFromStartPageToken(pageSize);
+ } else {
+ return new ReadEverythingPageToken();
+ }
+ } else {
+ // TODO implement, split out by the token's prefix
+ return new ReadEverythingPageToken();
+ }
+ }
+
+ /** Serialize a {@link PageToken} into a string */
+ @Override
+ public abstract String toString();
Review Comment:
Maybe nitpicky, but I'd prefer a method like `toStringExternal()` for clarity. `toString()` may return the same value, of course, but I think it is usually used "too lightly" so a dedicated serialization method should add some safety.
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/DonePageToken.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+public class DonePageToken extends PageToken {
+
+ public DonePageToken() {}
+
+ @Override
+ public String toString() {
+ return null;
+ }
+
+ @Override
+ protected PageToken updated(List<?> newData) {
+ return new DonePageToken();
Review Comment:
It is an error for a "done" token to have a next page, right? Should we throw an exception here?
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadFromStartPageToken.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation that has a page size, but no start offset. This can be used to
+ * represent a `limit`. When updated, it returns a token with the same semantics.
+ */
+public class ReadFromStartPageToken extends PageToken implements HasPageSize {
Review Comment:
nit: rename to `InitialPageToken`... not sure the name is readable :sweat_smile: but I'm trying to avoid confusion with a token to "Read From Start Page", which is what I tent to read it as :sweat_smile:
##########
service/common/build.gradle.kts:
##########
@@ -104,6 +104,7 @@ dependencies {
testFixturesImplementation(project(":polaris-api-management-model"))
testFixturesImplementation(project(":polaris-api-management-service"))
testFixturesImplementation(project(":polaris-api-iceberg-service"))
+ testFixturesImplementation(project(":polaris-jpa-model"))
Review Comment:
Is this still needed? why?
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ /** Build a new PageToken that reads everything */
+ public static PageToken readEverything() {
+ return build(null, null);
+ }
+
+ /** Build a new PageToken from an input String, without a specified page size */
+ public static PageToken fromString(String token) {
+ return build(token, null);
+ }
+
+ /** Build a new PageToken from a limit */
+ public static PageToken fromLimit(Integer pageSize) {
+ return build(null, pageSize);
+ }
+
+ /** Build a {@link PageToken} from the input string and page size */
+ public static PageToken build(String token, Integer pageSize) {
+ if (token == null || token.isEmpty()) {
+ if (pageSize != null) {
+ return new ReadFromStartPageToken(pageSize);
+ } else {
+ return new ReadEverythingPageToken();
+ }
+ } else {
+ // TODO implement, split out by the token's prefix
+ return new ReadEverythingPageToken();
Review Comment:
Maybe throw an `IllegalArgumentException` for now?
##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -169,6 +170,23 @@ public static boolean isCreate(UpdateTableRequest request) {
return isCreate;
}
+ public ListNamespacesResponse listNamespaces(
+ Namespace parent, String pageToken, Integer pageSize) {
+ PolarisAuthorizableOperation op = PolarisAuthorizableOperation.LIST_NAMESPACES;
+ authorizeBasicNamespaceOperationOrThrow(op, parent);
+
+ if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
+ Page<Namespace> results = polarisCatalog.listNamespaces(parent, pageToken, pageSize);
+ return ListNamespacesResponse.builder()
+ .addAll(results.items)
+ .nextPageToken(results.pageToken.toString())
+ .build();
+ } else {
+ return CatalogHandlers.listNamespaces(
+ namespaceCatalog, parent, pageToken, String.valueOf(pageSize));
Review Comment:
I believe we can do better in Polaris by using the REST API of `baseCatalog` directly.
This can be deferred.
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadFromStartPageToken.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation that has a page size, but no start offset. This can be used to
+ * represent a `limit`. When updated, it returns a token with the same semantics.
+ */
+public class ReadFromStartPageToken extends PageToken implements HasPageSize {
+
+ public static final String PREFIX = "read-from-start";
+
+ private final int pageSize;
+
+ public ReadFromStartPageToken(int pageSize) {
+ this.pageSize = pageSize;
+ }
+
+ @Override
+ public int getPageSize() {
+ return pageSize;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("%s/%d", PREFIX, pageSize);
+ }
+
+ @Override
+ protected PageToken updated(List<?> newData) {
+ return new ReadFromStartPageToken(pageSize);
Review Comment:
This looks incorrect to me. It will likely induce infinite loops in clients that try to do pagination.
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/DonePageToken.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+public class DonePageToken extends PageToken {
+
+ public DonePageToken() {}
+
+ @Override
+ public String toString() {
+ return null;
Review Comment:
nit: This is probably a case where we may want to have `toString() == "DONE"`, but `toStringExternal() == null`.
##########
extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java:
##########
@@ -450,27 +455,33 @@ public List<EntityNameLookupRecord> lookupEntityActiveBatchInCurrentTxn(
entity.getParentId(),
entity.getName(),
entity.getTypeCode(),
- entity.getSubTypeCode()));
+ entity.getSubTypeCode()),
+ pageToken);
}
@Override
- public @Nonnull <T> List<T> listEntitiesInCurrentTxn(
+ public @Nonnull <T> Page<T> listEntitiesInCurrentTxn(
@Nonnull PolarisCallContext callCtx,
long catalogId,
long parentId,
@Nonnull PolarisEntityType entityType,
- int limit,
@Nonnull Predicate<PolarisBaseEntity> entityFilter,
- @Nonnull Function<PolarisBaseEntity, T> transformer) {
+ @Nonnull Function<PolarisBaseEntity, T> transformer,
+ @Nonnull PageToken pageToken) {
// full range scan under the parent for that type
- return this.store
- .lookupFullEntitiesActive(localSession.get(), catalogId, parentId, entityType)
- .stream()
- .map(ModelEntity::toEntity)
- .filter(entityFilter)
- .limit(limit)
- .map(transformer)
- .collect(Collectors.toList());
+ Stream<PolarisBaseEntity> data =
+ this.store
+ .lookupFullEntitiesActive(
+ localSession.get(), catalogId, parentId, entityType, pageToken)
+ .stream()
+ .map(ModelEntity::toEntity)
+ .filter(entityFilter);
+
+ if (pageToken instanceof HasPageSize hasPageSize) {
Review Comment:
I'm not sure this if functionally correct. If the token is a sub-class of `HasPageSize` but has other information that the client provided, but EclipseLink does not recognise, I do not think it would be correct to just use the page size from the token.
I believe we should check for a specific type here and fail on all types, whose details cannot be fully processed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080097116
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ /** Build a new PageToken that reads everything */
+ public static PageToken readEverything() {
+ return build(null, null);
+ }
+
+ /** Build a new PageToken from an input String, without a specified page size */
+ public static PageToken fromString(String token) {
+ return build(token, null);
+ }
+
+ /** Build a new PageToken from a limit */
+ public static PageToken fromLimit(Integer pageSize) {
+ return build(null, pageSize);
+ }
+
+ /** Build a {@link PageToken} from the input string and page size */
+ public static PageToken build(String token, Integer pageSize) {
+ if (token == null || token.isEmpty()) {
+ if (pageSize != null) {
+ return new ReadFromStartPageToken(pageSize);
+ } else {
+ return new ReadEverythingPageToken();
+ }
+ } else {
+ // TODO implement, split out by the token's prefix
+ return new ReadEverythingPageToken();
Review Comment:
re: backward compatibility, clients are not supposed to "invent" page tokens, but Polaris in this PR still should never return a valid non-empty token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2073944331
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadFromStartPageToken.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * This is needed until real pagination is implemented just to capture limit / page-size without any
+ * particular start position TODO: Remove this type
Review Comment:
This entire type can get removed once the `pageTokenBuilder` implementations are updated to return a more useful `PageToken` implementation.
It's needed for now because we are updating interfaces to remove `int limit` in lieu of a `PageToken`, but `ReadEverythingPageToken` (which will stay for testing and for calls that want no pagination) doesn't support a limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078473013
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
For 1 I'd say we should use some server default page size for backends that support pagination. It may be large, but not unlimited.
For backends that do not support pagination, unlimited result set is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078533533
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
I see... I still think placing the logic (e.g. parsing the string representation) inside the PageToken implementation is nice encapsulation.
This specific comment is correct in that `fromStringComponents` in `ReadEverythingPageToken` looks a little funny, but if you look in `PageToken.fromString`, you'll see that we aren't just throwing out the components; we first check the length.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078608941
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
There are a couple of issues with that that I see:
1. We should like to be able to recognize an invalid page token way before we pass it to the persistence layer
2. Having unrelated types `OffsetToken(offset: int, page_size: int)` and `PageToken(token: String, page_size: int)` seems wrong
3. We might want to do something with the page token above the persistence layer -- for example client-side filtering or logging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080124909
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadFromStartPageToken.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation that has a page size, but no start offset. This can be used to
+ * represent a `limit`. When updated, it returns a token with the same semantics.
+ */
+public class ReadFromStartPageToken extends PageToken implements HasPageSize {
Review Comment:
Maybe `LimitPageToken`? `InitialPageToken` may be misleading because it implies there are subsequent page tokens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080131873
##########
service/common/build.gradle.kts:
##########
@@ -104,6 +104,7 @@ dependencies {
testFixturesImplementation(project(":polaris-api-management-model"))
testFixturesImplementation(project(":polaris-api-management-service"))
testFixturesImplementation(project(":polaris-api-iceberg-service"))
+ testFixturesImplementation(project(":polaris-jpa-model"))
Review Comment:
Sorry I missed this, good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080163100
##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -169,6 +170,23 @@ public static boolean isCreate(UpdateTableRequest request) {
return isCreate;
}
+ public ListNamespacesResponse listNamespaces(
+ Namespace parent, String pageToken, Integer pageSize) {
+ PolarisAuthorizableOperation op = PolarisAuthorizableOperation.LIST_NAMESPACES;
+ authorizeBasicNamespaceOperationOrThrow(op, parent);
+
+ if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
+ Page<Namespace> results = polarisCatalog.listNamespaces(parent, pageToken, pageSize);
+ return ListNamespacesResponse.builder()
+ .addAll(results.items)
+ .nextPageToken(results.pageToken.toString())
+ .build();
+ } else {
+ return CatalogHandlers.listNamespaces(
+ namespaceCatalog, parent, pageToken, String.valueOf(pageSize));
Review Comment:
Followup SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078508641
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
I don't think we want to duplicate the logic, since different persistence impls can definitely share the same PageToken impl. In fact, all 3 backends should really use EntityIdPageToken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078635592
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
In this case how about a parsing factory approach?
* Persistence returns whatever token fits best
* token has to `toStringExternal()` method that follows a common format with type codes
* There is one `TokenParser` class that knows about all the subtypes (may be an enum)
* the `parse` method uses the type code to choose the runtime java class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078347285
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/dao/entity/EntitiesResult.java:
##########
@@ -23,13 +23,21 @@
import jakarta.annotation.Nonnull;
import jakarta.annotation.Nullable;
import java.util.List;
+import java.util.Optional;
import org.apache.polaris.core.entity.PolarisBaseEntity;
+import org.apache.polaris.core.persistence.pagination.Page;
+import org.apache.polaris.core.persistence.pagination.PageToken;
/** a set of returned entities result */
public class EntitiesResult extends BaseResult {
// null if not success. Else the list of entities being returned
private final List<PolarisBaseEntity> entities;
+ private final Optional<PageToken> pageTokenOpt;
+
+ public static EntitiesResult fromPolarisPage(Page<PolarisBaseEntity> page) {
Review Comment:
agh, good catch! I put too much faith in my IDE when refactoring `PolarisPage` -> `Page`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078346191
##########
extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java:
##########
@@ -762,4 +769,11 @@ public void rollback() {
session.getTransaction().rollback();
}
}
+
+ // TODO support pagination
Review Comment:
Good point, let me update this comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078353024
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
It's a good question, let me briefly overview how it's intended to behave.
The user may provide a token, a page-size, both, or neither.
1. If they provide neither, there is no pagination.
2. If they provide just a page size, we start a new listing operation with the given page size
3. If they provide just a page token, we start from the location specified in the token, with the page size specified in the token.
4. If they provide both, we start from the location specified in the token, with the page size specified in the request.
So this method is useful for situations (2) and (4).
Also, good point re: null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078354642
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
Review Comment:
real answer: I think Immutables was not in the project when I first wrote this code :)
However, I do still prefer plain old java for things like this. We use this pattern through the Entities in Polaris. If there is a strong objection here, I can try to refactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078494703
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
How about `PageToken` being a thin wrapper around `<String token, int requestedPageSize>` and each Persistence impl will be in charge of parsing its own stuff. A shared class in `polaris-core` could be used for the common stuff, but ultimately each Persistence will be in control over the internals of page tokens.
`Page` would contain only the encoded token as a `String`... WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078881363
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
For at least the first iteration, I want to take a hard line on complying with the spec. Per the quote you cited that means that if there's no page token, we return everything.
Today, we just return all results no matter what the user asks for. I think it's okay to keep returning all results if the user asks for it, but at least now we will give users the option to ask for something else. If nothing else it's an incremental improvement.
wrt. null vs empty string, I'm a bit paranoid about us accidentally paginating results when the user doesn't expect it. So there too I think it's better to err on the side of the current (and still spec-compliant) behavior of just returning everything.
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
For at least the first iteration, I want to take a hard line on complying with the spec. Per the quote you cited that means that if there's no page token, we return everything.
Remember that today we just return all results no matter what the user asks for. I think it's okay to keep returning all results if the user asks for it, but at least now we will give users the option to ask for something else. If nothing else it's an incremental improvement.
wrt. null vs empty string, I'm a bit paranoid about us accidentally paginating results when the user doesn't expect it. So there too I think it's better to err on the side of the current (and still spec-compliant) behavior of just returning everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on PR #1528:
URL: https://github.com/apache/polaris/pull/1528#issuecomment-2861736691
Thanks @dimas-b for the thorough review! Based on your advice above I've refactored things quite a bit and seemingly simplified the interfaces greatly. Several types are gone and the way that we build `PageToken`s is much simpler now. Let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078282045
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/dao/entity/EntitiesResult.java:
##########
@@ -23,13 +23,21 @@
import jakarta.annotation.Nonnull;
import jakarta.annotation.Nullable;
import java.util.List;
+import java.util.Optional;
import org.apache.polaris.core.entity.PolarisBaseEntity;
+import org.apache.polaris.core.persistence.pagination.Page;
+import org.apache.polaris.core.persistence.pagination.PageToken;
/** a set of returned entities result */
public class EntitiesResult extends BaseResult {
// null if not success. Else the list of entities being returned
private final List<PolarisBaseEntity> entities;
+ private final Optional<PageToken> pageTokenOpt;
+
+ public static EntitiesResult fromPolarisPage(Page<PolarisBaseEntity> page) {
Review Comment:
Why "Polaris Page"? :)
##########
extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java:
##########
@@ -762,4 +769,11 @@ public void rollback() {
session.getTransaction().rollback();
}
}
+
+ // TODO support pagination
Review Comment:
I'd leave it out - EclipseLink is deprecated, the dev effort would be lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078594565
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
In a way, I guess my comments might also suggest a particular delegation model. Does the page token have to be transparent to all of Polaris code? Does it have meaning only inside a particular persistence implementation?... I tend to lead towards the latter.
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
In a way, I guess my comments might also suggest a particular delegation model. Does the page token have to be transparent to all of Polaris code? Does it have meaning only inside a particular persistence implementation?... I tend to lean towards the latter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080141520
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/DonePageToken.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+public class DonePageToken extends PageToken {
+
+ public DonePageToken() {}
+
+ @Override
+ public String toString() {
+ return null;
+ }
+
+ @Override
+ protected PageToken updated(List<?> newData) {
+ return new DonePageToken();
Review Comment:
I think so, but it'll depend how the caller uses it. I'll make the change for now and we can figure out the particulars of some of these methods when we actually implement pagination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080138610
##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -169,6 +170,23 @@ public static boolean isCreate(UpdateTableRequest request) {
return isCreate;
}
+ public ListNamespacesResponse listNamespaces(
+ Namespace parent, String pageToken, Integer pageSize) {
+ PolarisAuthorizableOperation op = PolarisAuthorizableOperation.LIST_NAMESPACES;
+ authorizeBasicNamespaceOperationOrThrow(op, parent);
+
+ if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
+ Page<Namespace> results = polarisCatalog.listNamespaces(parent, pageToken, pageSize);
+ return ListNamespacesResponse.builder()
+ .addAll(results.items)
+ .nextPageToken(results.pageToken.toString())
+ .build();
+ } else {
+ return CatalogHandlers.listNamespaces(
+ namespaceCatalog, parent, pageToken, String.valueOf(pageSize));
Review Comment:
Either way, I think we have to delegate pagination to the federated catalog if it supports pagination. `CatalogHandlers.listNamespaces` has a trivial implementation that is sub-optimal when the catalog is a REST Catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078509490
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
This is mostly governed by the Iceberg spec and our goal should be to comply with that.
What you wrote is true, that the server may return more or less records than the requested page size (whether embedded in the token or explicitly requested by the client). But we still need a way for the client to request a given page size as per the spec. The four scenarios outlined above are still valid and are still supported by the APIs defined here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078512378
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
to clarify: we can still have fewer page token representations than Persistence implementations, but the implementations do not have to be polymorphic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078473013
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
For 1 I'd say we should use some server default page size. It may be large, but not unlimited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078553909
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
Specs are not infallible :)
Quoting from the REST API [spec](https://github.com/apache/iceberg/blob/f7c6d57a03282516c6f988fa23a809416cd765b1/open-api/rest-catalog-open-api.yaml#L1699-L1701)
```
Servers that support pagination must return all results in a single response with the value
of `next-page-token` set to `null` if the query parameter `pageToken` is not set in the
request.
```
IMHO, this requirement is a recipe for disaster if, for example, a catalog has millions of entries.
I would not want to force Polaris into following this statement by the letter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080120839
##########
extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java:
##########
@@ -450,27 +455,33 @@ public List<EntityNameLookupRecord> lookupEntityActiveBatchInCurrentTxn(
entity.getParentId(),
entity.getName(),
entity.getTypeCode(),
- entity.getSubTypeCode()));
+ entity.getSubTypeCode()),
+ pageToken);
}
@Override
- public @Nonnull <T> List<T> listEntitiesInCurrentTxn(
+ public @Nonnull <T> Page<T> listEntitiesInCurrentTxn(
@Nonnull PolarisCallContext callCtx,
long catalogId,
long parentId,
@Nonnull PolarisEntityType entityType,
- int limit,
@Nonnull Predicate<PolarisBaseEntity> entityFilter,
- @Nonnull Function<PolarisBaseEntity, T> transformer) {
+ @Nonnull Function<PolarisBaseEntity, T> transformer,
+ @Nonnull PageToken pageToken) {
// full range scan under the parent for that type
- return this.store
- .lookupFullEntitiesActive(localSession.get(), catalogId, parentId, entityType)
- .stream()
- .map(ModelEntity::toEntity)
- .filter(entityFilter)
- .limit(limit)
- .map(transformer)
- .collect(Collectors.toList());
+ Stream<PolarisBaseEntity> data =
+ this.store
+ .lookupFullEntitiesActive(
+ localSession.get(), catalogId, parentId, entityType, pageToken)
+ .stream()
+ .map(ModelEntity::toEntity)
+ .filter(entityFilter);
+
+ if (pageToken instanceof HasPageSize hasPageSize) {
Review Comment:
This is correct for the current code, where `HasPageSize` is just used as a limit. When page tokens are implemented this logic will definitely change!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080133654
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/DonePageToken.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+public class DonePageToken extends PageToken {
+
+ public DonePageToken() {}
+
+ @Override
+ public String toString() {
+ return null;
Review Comment:
Oh, yea... good point. I shied away from `toTokenString` initially because I wanted to avoid accidental use of `toString` and the divergence of the two methods, but I think you're right that in this case we want divergence. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078361797
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java:
##########
@@ -408,4 +414,7 @@ boolean hasChildren(
default BasePersistence detach() {
return this;
}
+
+ @Nonnull
+ PageToken.PageTokenBuilder<?> pageTokenBuilder();
Review Comment:
The intent here is that different metastores could have different PageToken implementations that they need to use. For example, it seems like EclipseLink may just not support pagination, which is fine.
Practically, we need a way for persistence to be able to say "Ok, I fetched the first N results, here's a page token that will give you the next page". The builder which constructs that page token varies by persistence implementation. But if a different code structure would work better here I'm very open to it. Maybe it's more like a factory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078514633
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null || tokenString.isEmpty()) {
+ throw new IllegalArgumentException("Cannot build page token from empty string");
+ } else {
+ try {
+ String decoded =
+ new String(Base64.getDecoder().decode(tokenString), StandardCharsets.UTF_8);
+ String[] parts = decoded.split(":");
+
+ // +2 to account for the prefix and checksum.
+ if (parts.length != expectedComponents() + 2 || !parts[0].equals(tokenPrefix())) {
+ throw new IllegalArgumentException("Invalid token format in token: " + tokenString);
+ }
+
+ // Cut off prefix and checksum
+ T result = fromStringComponents(Arrays.asList(parts).subList(1, parts.length - 1));
+ result.validate();
+ return result;
+ } catch (RuntimeException e) {
+ throw new IllegalArgumentException("Invalid page token: " + tokenString, e);
+ }
+ }
+ }
+
+ /** Construct a {@link PageToken} from a plain limit */
+ public final PageToken fromLimit(Integer limit) {
+ if (limit == null) {
+ return ReadEverythingPageToken.get();
+ } else {
+ return fromLimitImpl(limit);
+ }
+ }
+
+ /** Construct a {@link PageToken} from a plain limit */
+ protected abstract T fromLimitImpl(int limit);
+
+ /**
+ * {@link PageTokenBuilder} implementations should implement this to build a {@link PageToken}
+ * from components in a string token. These components should be the same ones returned by
+ * {@link #getComponents()} and won't include the token prefix or the checksum.
+ */
+ protected abstract T fromStringComponents(List<String> components);
+ }
+
+ /** Convert this into components that the serialized token string will be built from. */
+ protected abstract List<String> getComponents();
+
+ /**
+ * Builds a new page token to reflect new data that's been read. If the amount of data read is
+ * less than the pageSize, this will return {@link PageToken#END}(null)
+ */
+ protected abstract PageToken updated(List<?> newData);
+
+ /**
+ * Builds a {@link Page <T>} from a {@link List<T>}. The {@link PageToken} attached to the new
+ * {@link Page <T>} is the same as the result of calling {@link #updated(List)} on this {@link
+ * PageToken}.
+ */
+ public final <T> Page<T> buildNextPage(List<T> data) {
+ return new Page<T>(updated(data), data);
+ }
+
+ /**
+ * Return a new {@link PageToken} with an updated page size. If the pageSize provided is null, the
+ * existing page size will be preserved.
+ */
+ public abstract PageToken withPageSize(Integer pageSize);
+
+ /** Serialize a {@link PageToken} into a string */
+ @Override
+ public final String toString() {
+ List<String> components = getComponents();
+ String prefix = getBuilder().tokenPrefix();
+ String componentString = String.join(":", components);
+ String checksum = String.valueOf(componentString.hashCode());
+ List<String> allElements =
+ Stream.of(prefix, componentString, checksum)
+ .filter(s -> !s.isEmpty())
+ .collect(Collectors.toList());
+ String rawString = String.join(":", allElements);
+ return Base64.getEncoder().encodeToString(rawString.getBytes(StandardCharsets.UTF_8));
Review Comment:
`getUrlEncoder` might be preferable.
##########
service/common/build.gradle.kts:
##########
@@ -29,6 +29,7 @@ dependencies {
implementation(project(":polaris-api-management-service"))
implementation(project(":polaris-api-iceberg-service"))
implementation(project(":polaris-api-catalog-service"))
+ implementation(project(":polaris-jpa-model"))
Review Comment:
A dependency from `service/common` to JPA does not look right to me. When EclipseLink is removed, JPA will be removed too, I guess.
##########
service/common/src/main/java/org/apache/polaris/service/types/ListNamespacesResponseWithPageToken.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.types;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import com.google.common.base.Preconditions;
+import java.util.List;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.rest.responses.ListNamespacesResponse;
+import org.apache.polaris.core.persistence.pagination.Page;
+import org.apache.polaris.core.persistence.pagination.PageToken;
+
+/**
+ * Used in lieu of {@link ListNamespacesResponse} when there may be a {@link PageToken} associated
+ * with the response. Callers can use this {@link PageToken} to continue the listing operation and
+ * obtain more results.
+ */
+public class ListNamespacesResponseWithPageToken extends ListNamespacesResponse {
Review Comment:
Why do we need to sub-class? `ListNamespacesResponse` already has a `nextPageToken` property :thinking:
##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -169,6 +173,19 @@ public static boolean isCreate(UpdateTableRequest request) {
return isCreate;
}
+ public ListNamespacesResponseWithPageToken listNamespaces(Namespace parent, PageToken pageToken) {
+ PolarisAuthorizableOperation op = PolarisAuthorizableOperation.LIST_NAMESPACES;
+ authorizeBasicNamespaceOperationOrThrow(op, parent);
+
+ if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
+ return ListNamespacesResponseWithPageToken.fromPage(
+ polarisCatalog.listNamespaces(parent, pageToken));
+ } else {
+ return ListNamespacesResponseWithPageToken.fromPage(
+ Page.fromData(CatalogHandlers.listNamespaces(namespaceCatalog, parent).namespaces()));
Review Comment:
Should we not be using the page token from the Federated RESTCatalog in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078635592
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
In this case how about a parsing factory approach?
* Persistence returns whatever token they fits best
* token has to `toStringExternal()` method that follows a common format with type codes
* There is one `TokenParser` class that knows about all the subtypes (may be an enum)
* the `parse` method uses the type code to choose the runtime java class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078882308
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
For now, I'll do away with the idea of different metastores needing different page tokens and we can come back to this idea if the need for this every actually comes about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078353024
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
It's a good question, let me briefly overview how it's intended to behave.
The user may provide a token, a page-size, both, or neither.
1. If they provide neither, there is no pagination.
2. If they provide just a page size, we start a new listing operation with the given page size
3. If they provide just a page token, we start from the location specified in the token, with the page size specified in the token.
4. If they provide both, we start from the location specified in the token, with the page size specified in the request.
So this method is useful for situations (2) and (4). Consider, too, the case where the user does (4), but then follows up with (3) -- I think they would expect the same page size to continue to apply in the second call, which is why we stick the page size in the token itself.
Also, good point re: null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078546138
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
We can certainly have a hierarchy of page token implementations and share common code. My idea is that the entry into the parsing logic does not have to be polymorphic. For example, JDBC Persistence uses an "offset page token", so it'd call `OffsetToken t = OffsetToken.parse(String)`. A NoSQL impl might use a "previous name in index" token, it would call `NameToken t = NameToken.parse(String)`. Internally both `OffsetToken` and `NameToken` could extend a common base class, or simply share common static utility methods.
The key here is that input is converted to a simple `PageToken(String, int)` without parsing. Parsing happens at the persistence layer where we know the expected token type.
If we ever have to change token encoding, the parsing methods can still return the appropriate sub-class by analysing the token structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078881363
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
For at least the first iteration, I want to take a hard line on complying with the spec. Today, we just return all results no matter what the user asks for. I think it's okay to keep returning all results if the user asks for it, but at least now we will give users the option to ask for something else.
wrt. null vs empty string, I'm a bit paranoid about us accidentally paginating results when the user doesn't expect it. So I think it's better to err on the side of the current (and still spec-compliant) behavior of just returning everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080136331
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadFromStartPageToken.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation that has a page size, but no start offset. This can be used to
+ * represent a `limit`. When updated, it returns a token with the same semantics.
+ */
+public class ReadFromStartPageToken extends PageToken implements HasPageSize {
Review Comment:
`LimitPageToken` LGTM :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080132406
##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -169,6 +170,23 @@ public static boolean isCreate(UpdateTableRequest request) {
return isCreate;
}
+ public ListNamespacesResponse listNamespaces(
+ Namespace parent, String pageToken, Integer pageSize) {
+ PolarisAuthorizableOperation op = PolarisAuthorizableOperation.LIST_NAMESPACES;
+ authorizeBasicNamespaceOperationOrThrow(op, parent);
+
+ if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
+ Page<Namespace> results = polarisCatalog.listNamespaces(parent, pageToken, pageSize);
+ return ListNamespacesResponse.builder()
+ .addAll(results.items)
+ .nextPageToken(results.pageToken.toString())
+ .build();
+ } else {
+ return CatalogHandlers.listNamespaces(
+ namespaceCatalog, parent, pageToken, String.valueOf(pageSize));
Review Comment:
It may not be a REST catalog :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2073950129
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
Review Comment:
Reviving the discussion on the name below...
It's currently called `PageToken` because it encapsulates the data in `page-token` sent by the client. It also encapsulates `page-size` and so there has been some objection to the naming.
At its core, this `PageToken` is used to describe how persistence should build a page of data. It is, by itself, not a sufficient "page request" in that it doesn't contain information like the type of entity to search for. But it is used in conjunction to with a request to filter results down.
Because the Iceberg spec is vague about the `page-token` implementation -- calling it "opaque" -- I feel that a page token which contains information about the size of the page is still consistent with the spec. `page-size` is still respected when it's present in a request alongside `page-token`.
To be clear, it's not just the Java type that contains the page size. The string `page-token` itself contains the page size information and the Java type is reflecting that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078494703
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
How about `PageToken` being a thin wrapper around `<String token, int requestedPageSize>` and each Persistence impl will be in charge of parsing its own stuff. A shared class in `polaris-core` could be used for the common stuff, but ultimately each Persistence will be in control over the internal of page tokens.
`Page` would contain only the encoded token as a `String`... WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2078288730
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
Review Comment:
Why not leverage generated builders by using `@PolarisImmutable`?
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
Is `null` that much different from an empty string?
Since the intention is to convey both the "page pointer" and the "page size" in `PageToken`, why have have a `from(@Nullable String token, int pageSize)` method?
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java:
##########
@@ -408,4 +414,7 @@ boolean hasChildren(
default BasePersistence detach() {
return this;
}
+
+ @Nonnull
+ PageToken.PageTokenBuilder<?> pageTokenBuilder();
Review Comment:
minor: Builders are usually quite simple collectors of programmatically collected arguments. Here it looks like this is more of a "parser". I'd personally prefer builders to be generated (immutables), and have custom parser factory interface.
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/ReadEverythingPageToken.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+
+/**
+ * A {@link PageToken} implementation for readers who want to read everything. The behavior when
+ * using this token should be the same as when reading without a token.
+ */
+public class ReadEverythingPageToken extends PageToken {
+
+ private ReadEverythingPageToken() {
+ this.pageSize = Integer.MAX_VALUE;
+ validate();
+ }
+
+ /** Get a {@link ReadEverythingPageToken} */
+ public static ReadEverythingPageToken get() {
+ return new ReadEverythingPageToken();
+ }
+
+ public static PageTokenBuilder<ReadEverythingPageToken> builder() {
+ return new ReadEverythingPageTokenBuilder();
+ }
+
+ @Override
+ protected PageTokenBuilder<?> getBuilder() {
+ return builder();
+ }
+
+ /** A {@link PageTokenBuilder} implementation for {@link ReadEverythingPageToken} */
+ public static class ReadEverythingPageTokenBuilder
+ extends PageTokenBuilder<ReadEverythingPageToken> {
+
+ private ReadEverythingPageTokenBuilder() {}
+
+ @Override
+ public String tokenPrefix() {
+ return "polaris-read-everything";
+ }
+
+ @Override
+ public int expectedComponents() {
+ return 0;
+ }
+
+ @Override
+ protected ReadEverythingPageToken fromStringComponents(List<String> components) {
+ return ReadEverythingPageToken.get();
Review Comment:
IMHO this is a skew in builder API and another point to my comment about "parser interface". The method signature reads as if components always matter, but they are ignored here.
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
+ } else {
+ return fromLimit(DEFAULT_PAGE_SIZE);
+ }
+ } else {
+ try {
+ String decoded =
+ new String(Base64.getDecoder().decode(tokenString), StandardCharsets.UTF_8);
+ String[] parts = decoded.split(":");
+
+ // +2 to account for the prefix and checksum.
+ if (parts.length != expectedComponents() + 2 || !parts[0].equals(tokenPrefix())) {
+ throw new IllegalArgumentException("Invalid token format in token: " + tokenString);
+ }
+
+ // Cut off prefix and checksum
+ T result = fromStringComponents(Arrays.asList(parts).subList(1, parts.length - 1));
+ result.validate();
+ return result;
+ } catch (Exception e) {
+ throw new IllegalArgumentException("Failed to decode page token: " + tokenString, e);
Review Comment:
nit: `Invalid page token: ...` (It's the client's fault, not server's)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Interface changes for pagination [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1528:
URL: https://github.com/apache/polaris/pull/1528#discussion_r2080093479
##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Represents a page token that can be used by operations like `listTables`. Clients that specify a
+ * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
+ * which is a serialized PageToken.
+ *
+ * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
+ * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
+ * response, that means there is no more data to read.
+ */
+public abstract class PageToken {
+
+ public int pageSize;
+
+ public static final PageToken END = null;
+ public static final int DEFAULT_PAGE_SIZE = 1000;
+
+ protected void validate() {
+ if (pageSize <= 0) {
+ throw new IllegalArgumentException("Page size must be greater than zero");
+ }
+ }
+
+ /**
+ * Get a new PageTokenBuilder from a PageToken. The PageTokenBuilder type should match the
+ * PageToken type. Implementations may also provide a static `builder` method to obtain the same
+ * PageTokenBuilder.
+ */
+ protected abstract PageTokenBuilder<?> getBuilder();
+
+ /** Allows `PageToken` implementations to implement methods like `fromLimit` */
+ public abstract static class PageTokenBuilder<T extends PageToken> {
+
+ /**
+ * A prefix that tokens are expected to start with, ideally unique across `PageTokenBuilder`
+ * implementations.
+ */
+ public abstract String tokenPrefix();
+
+ /**
+ * The number of expected components in a token. This should match the number of components
+ * returned by getComponents and shouldn't account for the prefix or the checksum.
+ */
+ public abstract int expectedComponents();
+
+ /** Deserialize a string into a {@link PageToken} */
+ public final PageToken fromString(String tokenString) {
+ if (tokenString == null) {
+ throw new IllegalArgumentException("Cannot build page token from null string");
+ } else if (tokenString.isEmpty()) {
+ if (this instanceof ReadEverythingPageToken.ReadEverythingPageTokenBuilder) {
+ return ReadEverythingPageToken.get();
Review Comment:
Fair point about minimizing disturbance to existing Polaris behaviour in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org