You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "dimas-b (via GitHub)" <gi...@apache.org> on 2025/06/09 18:27:05 UTC

[PR] Pagination alt [polaris]

dimas-b opened a new pull request, #1838:
URL: https://github.com/apache/polaris/pull/1838

   <!--
       Possible security vulnerabilities: STOP here and contact security@apache.org instead!
   
       Please update the title of the PR with a meaningful message - do not leave it "empty" or "generated"
       Please update this summary field:
   
       The summary should cover these topics, if applicable:
       * the motivation for the change
       * a description of the status quo, for example the current behavior
       * the desired behavior
       * etc
   
       PR checklist:
       - Do a self-review of your code before opening a pull request
       - Make sure that there's good test coverage for the changes included in this PR
       - Run tests locally before pushing a PR (./gradlew check)
       - Code should have comments where applicable. Particularly hard-to-understand
         areas deserve good in-line documentation.
       - Include changes and enhancements to the documentation (in site/content/in-dev/unreleased)
       - For Work In Progress Pull Requests, please use the Draft PR feature.
   
       Make sure to add the information BELOW this comment.
       Everything in this comment will NOT be added to the PR description.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2145668181


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPaging.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+/** Utility class for processing pagination of streams of entities ordered by some ID property. */
+public final class EntityIdPaging {

Review Comment:
   > but that is something to be done if and when we need it
   
   We have a (smaller) existing PR in #1555 that supports this use case, so I have some reservations about an alternative design that loses this functionality 



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPaging.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+/** Utility class for processing pagination of streams of entities ordered by some ID property. */
+public final class EntityIdPaging {

Review Comment:
   > but that is something to be done if and when we need it
   
   We have an existing PR in #1555 that supports this use case, so I have some reservations about an alternative design that loses this functionality 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2143220314


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;

Review Comment:
   Since we're defining the serde/deserde of the page token now, let's do it in a way that captures future page tokens we think are reasonable. I think that includes page requests with more than 2 discrete pieces of information.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140656499


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,82 +18,85 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import static com.google.common.base.Preconditions.checkState;
 
-/**
- * Represents a page token that can be used by operations like `listTables`. Clients that specify a
- * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
- * which is a serialized PageToken.
- *
- * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
- * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
- * response, that means there is no more data to read.
- */
-public abstract class PageToken {
+import jakarta.annotation.Nullable;
 
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageToken {
+  private final @Nullable String encodedDataReference;
+  private final int pageSize;
 
-  /** Build a new PageToken from an input String, without a specified page size */
-  public static PageToken fromString(String token) {
-    return build(token, null);
+  PageToken(@Nullable String encodedDataReference, int pageSize) {
+    this.encodedDataReference = encodedDataReference;
+    this.pageSize = pageSize;
   }
 
-  /** Build a new PageToken from a limit */
-  public static PageToken fromLimit(Integer pageSize) {
-    return build(null, pageSize);
+  /** Represents a non-paginated request. */
+  public static PageToken readEverything() {
+    return new PageToken(null, -1);
   }
 
-  /** Build a {@link PageToken} from the input string and page size */
-  public static PageToken build(String token, Integer pageSize) {
-    if (token == null || token.isEmpty()) {
-      if (pageSize != null) {
-        return new LimitPageToken(pageSize);
-      } else {
-        return new ReadEverythingPageToken();
-      }
-    } else {
-      // TODO implement, split out by the token's prefix
-      throw new IllegalArgumentException("Unrecognized page token: " + token);
-    }
+  /** Represents a request to start paginating with a particular page size. */
+  public static PageToken fromLimit(int limit) {
+    return new PageToken(null, limit);
   }
 
-  /** Serialize a {@link PageToken} into a string */
-  public abstract String toTokenString();
+  /**
+   * Reconstructs a page token from the API-level page token string (returned to the client in the
+   * response to a previous request for similar data) and an API-level new requested page size.

Review Comment:
   "API-level" here means any round trip to a client (specifically the Iceberg REST Catalog API).
   
   Do you have a suggestion for improving this javadoc text?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2144152233


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item

Review Comment:
   How about keeping current `PageToken` class mostly as it is, but replacing its `String encodedDataReference` with `DataPointer dataPointer`?
   
   It does make the code nicer by having a specific type, but IMHO it adds a difficulty from the processing perspective:
   * Parsing `DataPointer` to a specific sub-type when `PageToken` is constructed will require encoding type information in the token. That is redundant IMHO, but I'm ok with that change.
   * Persistence methods that need to "understand" a `DataPointer` will have to do type casts, which is not elegant, IMHO. Currently they do not have to, persistence implementations currently just call a known parse method, which then will raise exceptions if the data is malformed.
   
   As a middle ground option WDYT about using `DataPointer` to wrap the encoded `String` (no sub-types), but still use static parsing methods to be called from Persistence?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2144159465


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;
+    if (requestedPageToken != null) {
+      byte[] bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      String token = new String(bytes, StandardCharsets.UTF_8);
+      int idx = token.indexOf(':');
+      checkArgument(idx > 0, "Invalid page token: %s", requestedPageToken);
+      int prevPageSize = Integer.parseInt(token.substring(0, idx));
+      encodedDataReference = token.substring(idx + 1);
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(prevPageSize);
+    } else {
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(-1);
+    }
+
+    return new PageToken(encodedDataReference, pageSize);
+  }
+
+  static @Nullable String encodePageToken(

Review Comment:
   `decodePageRequest` is called from `PageToken` when the token is received, but `encodePageToken` is called from`Page` when it is returned to the client. This class is the place to share code between `PageToken` and `Page`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2145640811


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item

Review Comment:
   I don't think just wrapping the encoded string is an adequate Java representation of the `page-token` -- we should extract the information from that encoded string.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2145648492


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;

Review Comment:
   I think this thread is just about not hard-coding the number of components in a page token right? Yes, we could add different encode/decode methods, but this one in `PageTokenUtil` looks like the canonical one in this design



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2145659867


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,82 +18,85 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import static com.google.common.base.Preconditions.checkState;
 
-/**
- * Represents a page token that can be used by operations like `listTables`. Clients that specify a
- * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
- * which is a serialized PageToken.
- *
- * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
- * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
- * response, that means there is no more data to read.
- */
-public abstract class PageToken {
+import jakarta.annotation.Nullable;
 
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageToken {
+  private final @Nullable String encodedDataReference;
+  private final int pageSize;
 
-  /** Build a new PageToken from an input String, without a specified page size */
-  public static PageToken fromString(String token) {
-    return build(token, null);
+  PageToken(@Nullable String encodedDataReference, int pageSize) {
+    this.encodedDataReference = encodedDataReference;
+    this.pageSize = pageSize;
   }
 
-  /** Build a new PageToken from a limit */
-  public static PageToken fromLimit(Integer pageSize) {
-    return build(null, pageSize);
+  /** Represents a non-paginated request. */
+  public static PageToken readEverything() {
+    return new PageToken(null, -1);
   }
 
-  /** Build a {@link PageToken} from the input string and page size */
-  public static PageToken build(String token, Integer pageSize) {
-    if (token == null || token.isEmpty()) {
-      if (pageSize != null) {
-        return new LimitPageToken(pageSize);
-      } else {
-        return new ReadEverythingPageToken();
-      }
-    } else {
-      // TODO implement, split out by the token's prefix
-      throw new IllegalArgumentException("Unrecognized page token: " + token);
-    }
+  /** Represents a request to start paginating with a particular page size. */
+  public static PageToken fromLimit(int limit) {
+    return new PageToken(null, limit);
   }
 
-  /** Serialize a {@link PageToken} into a string */
-  public abstract String toTokenString();
+  /**
+   * Reconstructs a page token from the API-level page token string (returned to the client in the
+   * response to a previous request for similar data) and an API-level new requested page size.

Review Comment:
   Isn't it just "a page token string"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140598351


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+public class EntityIdPageToken implements PageToken {
+  private final boolean hasMore;
+  private final boolean hasPrevious;
+  private final long entityId;
+
+  private EntityIdPageToken(boolean hasMore, boolean hasPrevious, long entityId) {
+    this.hasMore = hasMore;
+    this.hasPrevious = hasPrevious;
+    this.entityId = entityId;
+  }
+
+  public static EntityIdPageToken forPage(List<PolarisBaseEntity> data) {

Review Comment:
   reimplemented



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140591476


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import jakarta.annotation.Nullable;
+
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageRequest {
+  private final @Nullable String innerPageToken;
+  private final int pageSize;
+
+  PageRequest(@Nullable String innerPageToken, int pageSize) {
+    this.innerPageToken = innerPageToken;
+    this.pageSize = pageSize;
+  }
+
+  /** Represents a non-paginated request. */
+  public static PageRequest readEverything() {
+    return new PageRequest(null, -1);
+  }
+
+  /** Represents a request to start paginating with a particular page size. */
+  public static PageRequest firstPage(int limit) {

Review Comment:
   This is now `PageToken.fromLimit()` (same name as before).
   
   The use case is for handling initial paged requests where a size is provided, but a token in not. For API-based calls that is taken care of by the `decode()` method. This method is a convenience entry point for internal calls.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139134323


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item
+   *     from the previous page. The output of this function will be available from {@link
+   *     PageToken#encodedDataReference()} associated with the request for the next page.
+   */
+  public static <R, T> Page<R> mapped(
+      PageToken request, Stream<T> items, Function<T, R> mapper, Function<T, String> dataPointer) {
+    List<R> data;
+    if (request.paginationRequested()) {
+      data = new ArrayList<>(request.pageSize());
+    } else {
+      data = new ArrayList<>();
+    }
+
+    T last = null;
+    Iterator<T> it = iterator(items.spliterator());
+    while (it.hasNext() && (!request.paginationRequested() || data.size() < request.pageSize())) {
+      last = it.next();
+      data.add(mapper.apply(last));
+    }
+
+    if (request.paginationRequested() && data.size() < request.pageSize()) {
+      // the page was not filled, inform the client that there's no next page
+      last = null;
+    }
+
+    return new Page<>(request, dataPointer.apply(last), data);
+  }
+
+  public List<T> items() {
+    return items;
+  }
+
+  /**
+   * Returns a page token in encoded form suitable for returning to API clients. The string returned
+   * from this method is expected to be parsed by {@link PageToken#decode(String, Integer)} when
+   * servicing the request for the next page of related data.
+   */
+  public @Nullable String encodedResponseToken() {
+    return PageTokenUtil.encodePageToken(request, encodedDataReference);

Review Comment:
   Also, this looks like quite a thin method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139125285


##########
polaris-core/src/testFixtures/java/org/apache/polaris/core/persistence/PolarisTestMetaStoreManager.java:
##########
@@ -791,7 +793,8 @@ void dropEntity(List<PolarisEntityCore> catalogPath, PolarisBaseEntity entityToD
                       PolarisEntityType.CATALOG_ROLE,
                       PolarisEntitySubType.ANY_SUBTYPE,
                       PageToken.readEverything())
-                  .getEntities();
+                  .getEntities()
+                  .items();

Review Comment:
   This doesn't look necessary; the entities _are_ the items



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139123379


##########
polaris-core/src/main/java/org/apache/polaris/core/catalog/PolarisCatalogHelpers.java:
##########
@@ -62,20 +61,28 @@ public static Namespace getParentNamespace(Namespace namespace) {
     return Namespace.of(parentLevels);
   }
 
-  public static List<Namespace> nameAndIdToNamespaces(
-      List<PolarisEntity> catalogPath, List<PolarisEntity.NameAndId> entities) {
+  public static Namespace nameAndIdToNamespace(

Review Comment:
   Why is this needed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140643928


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item
+   *     from the previous page. The output of this function will be available from {@link
+   *     PageToken#encodedDataReference()} associated with the request for the next page.
+   */
+  public static <R, T> Page<R> mapped(
+      PageToken request, Stream<T> items, Function<T, R> mapper, Function<T, String> dataPointer) {
+    List<R> data;
+    if (request.paginationRequested()) {
+      data = new ArrayList<>(request.pageSize());
+    } else {
+      data = new ArrayList<>();
+    }
+
+    T last = null;
+    Iterator<T> it = iterator(items.spliterator());
+    while (it.hasNext() && (!request.paginationRequested() || data.size() < request.pageSize())) {
+      last = it.next();
+      data.add(mapper.apply(last));
+    }
+
+    if (request.paginationRequested() && data.size() < request.pageSize()) {
+      // the page was not filled, inform the client that there's no next page
+      last = null;
+    }
+
+    return new Page<>(request, dataPointer.apply(last), data);
+  }
+
+  public List<T> items() {
+    return items;
+  }
+
+  /**
+   * Returns a page token in encoded form suitable for returning to API clients. The string returned
+   * from this method is expected to be parsed by {@link PageToken#decode(String, Integer)} when
+   * servicing the request for the next page of related data.
+   */
+  public @Nullable String encodedResponseToken() {

Review Comment:
   Because encoding happens on the way "out", decoding on the way "in" (when handling REST API).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140648964


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;
+    if (requestedPageToken != null) {

Review Comment:
   I did. Is that a problem?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140778316


##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -798,13 +790,12 @@ private Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)
                 PolarisEntityType.NAMESPACE,
                 PolarisEntitySubType.NULL_SUBTYPE,
                 pageToken);
-    List<PolarisEntity.NameAndId> entities =
-        PolarisEntity.toNameAndIdList(listResult.getEntities());
-    List<Namespace> namespaces = PolarisCatalogHelpers.nameAndIdToNamespaces(catalogPath, entities);
     return listResult
-        .getPageToken()
-        .map(token -> new Page<>(token, namespaces))
-        .orElseGet(() -> Page.fromItems(namespaces));
+        .getEntities()
+        .map(
+            record ->

Review Comment:
   `record` comes from the type name of `EntityNameLookupRecord`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Delineate pagination requests and tokens [polaris]

Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2137576089


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.util.Optional;
+
+public final class PageTokenUtil {

Review Comment:
   This can become package-protected. Only non-package reference is from the unused function IcebergCatalog.buildPageRequest



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import jakarta.annotation.Nullable;
+
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageRequest {
+  private final @Nullable String innerPageToken;
+  private final int pageSize;
+
+  PageRequest(@Nullable String innerPageToken, int pageSize) {
+    this.innerPageToken = innerPageToken;
+    this.pageSize = pageSize;
+  }
+
+  /** Represents a non-paginated request. */
+  public static PageRequest readEverything() {
+    return new PageRequest(null, -1);
+  }
+
+  /** Represents a request to start paginating with a particular page size. */
+  public static PageRequest firstPage(int limit) {

Review Comment:
   What's the general use case of this one? It's only used in some `drop*()` _implementation_.



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,84 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
  * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
  * continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageRequest request;
+  private final PageToken nextToken;
+  private final List<T> items;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageRequest request, PageToken nextToken, List<T> items) {
+    this.request = request;
+    this.nextToken = nextToken;
     this.items = items;
   }
 
-  /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
-   */
-  public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+  public static <T> Page<T> allItems(List<T> items) {
+    return new Page<>(PageRequest.readEverything(), () -> null, items);
+  }
+
+  public static <T> Page<T> fromStream(

Review Comment:
   This one is only called from tests. Maybe remove this one and rename `mapped` to `fromStream`?



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -30,70 +29,8 @@
  * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
  * response, that means there is no more data to read.
  */
-public abstract class PageToken {
-
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
-
-  /** Build a new PageToken from an input String, without a specified page size */
-  public static PageToken fromString(String token) {
-    return build(token, null);
-  }
-
-  /** Build a new PageToken from a limit */
-  public static PageToken fromLimit(Integer pageSize) {
-    return build(null, pageSize);
-  }
-
-  /** Build a {@link PageToken} from the input string and page size */
-  public static PageToken build(String token, Integer pageSize) {
-    if (token == null || token.isEmpty()) {
-      if (pageSize != null) {
-        return new LimitPageToken(pageSize);
-      } else {
-        return new ReadEverythingPageToken();
-      }
-    } else {
-      // TODO implement, split out by the token's prefix
-      throw new IllegalArgumentException("Unrecognized page token: " + token);
-    }
-  }
-
+public abstract interface PageToken {

Review Comment:
   ```suggestion
   public interface PageToken {
   ```
   nit: superfluous keyword



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,8 +18,7 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import jakarta.annotation.Nullable;
 
 /**
  * Represents a page token that can be used by operations like `listTables`. Clients that specify a

Review Comment:
   Looks like the javadoc needs to be adapted.



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+public class EntityIdPageToken implements PageToken {
+  private final boolean hasMore;
+  private final boolean hasPrevious;
+  private final long entityId;
+
+  private EntityIdPageToken(boolean hasMore, boolean hasPrevious, long entityId) {
+    this.hasMore = hasMore;
+    this.hasPrevious = hasPrevious;
+    this.entityId = entityId;
+  }
+
+  public static EntityIdPageToken forPage(List<PolarisBaseEntity> data) {
+    if (data.isEmpty()) {
+      return forLastItem(null);
+    }
+
+    return forLastItem(data.get(data.size() - 1));
+  }
+
+  public static EntityIdPageToken forLastItem(PolarisBaseEntity entity) {

Review Comment:
   Shouldn't the parameter be `@Nonnull`, as the function name says `forLastItem`?



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+public class EntityIdPageToken implements PageToken {
+  private final boolean hasMore;
+  private final boolean hasPrevious;
+  private final long entityId;
+
+  private EntityIdPageToken(boolean hasMore, boolean hasPrevious, long entityId) {
+    this.hasMore = hasMore;
+    this.hasPrevious = hasPrevious;
+    this.entityId = entityId;
+  }
+
+  public static EntityIdPageToken forPage(List<PolarisBaseEntity> data) {
+    if (data.isEmpty()) {
+      return forLastItem(null);
+    }
+
+    return forLastItem(data.get(data.size() - 1));
+  }
+
+  public static EntityIdPageToken forLastItem(PolarisBaseEntity entity) {
+    if (entity == null) {
+      return new EntityIdPageToken(false, false, 0);
+    }
+    return new EntityIdPageToken(true, true, entity.getId());
+  }
+
+  /**
+   * Extracts an {@link EntityIdPageToken} from a {@link PageRequest} assuming the request is a
+   * continuation of the API call that previously produced an {@link EntityIdPageToken}.
+   */
+  public static EntityIdPageToken fromPageRequest(PageRequest pageRequest) {
+    String token = pageRequest.innerPageToken();
+    if (token == null) {
+      return new EntityIdPageToken(false, false, 0);
+    } else {
+      return new EntityIdPageToken(false, true, Long.parseLong(token));
+    }
+  }
+
+  public long getId() {
+    return entityId;
+  }
+
+  public boolean hasPrevious() {

Review Comment:
   Not sure I understand the meaning of `hasPrevious`.
   Doesn't the presence of a page-token implicitly mean that there's something "before"?



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.util.Optional;
+
+public final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  public static PageRequest decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String token = null;
+    if (requestedPageToken != null) {
+      int idx = requestedPageToken.indexOf(':');

Review Comment:
   I know it's not a new thing to this PR, but I don't understand why the size of the _previous_ page matters at all.
   AFAIK the page-size is included in every request. Feels a bit weird to have an "outer" and "inner" meaning of an otherwise opaque token.



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+public class EntityIdPageToken implements PageToken {
+  private final boolean hasMore;

Review Comment:
   Never used



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+public class EntityIdPageToken implements PageToken {
+  private final boolean hasMore;
+  private final boolean hasPrevious;
+  private final long entityId;
+
+  private EntityIdPageToken(boolean hasMore, boolean hasPrevious, long entityId) {
+    this.hasMore = hasMore;
+    this.hasPrevious = hasPrevious;
+    this.entityId = entityId;
+  }
+
+  public static EntityIdPageToken forPage(List<PolarisBaseEntity> data) {

Review Comment:
   This one's unused



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import jakarta.annotation.Nullable;
+
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageRequest {
+  private final @Nullable String innerPageToken;
+  private final int pageSize;
+
+  PageRequest(@Nullable String innerPageToken, int pageSize) {
+    this.innerPageToken = innerPageToken;
+    this.pageSize = pageSize;
+  }
+
+  /** Represents a non-paginated request. */
+  public static PageRequest readEverything() {
+    return new PageRequest(null, -1);
+  }
+
+  /** Represents a request to start paginating with a particular page size. */
+  public static PageRequest firstPage(int limit) {
+    return new PageRequest(null, limit);
+  }
+
+  /**
+   * Reconstructs a page request from the previous API-level page token and an API-level new
+   * requested page size.
+   *
+   * @param previousPageToken page token from the {@link Page#encodedResponseToken() previous page}
+   * @param requestedPageSize optional page size for the next page. If not set, the page size of the
+   *     previus page (encoded in the previous page token) will be reused.
+   */
+  public static PageRequest nextPage(

Review Comment:
   The function name and phrase 'from the previous API-level...` are a bit confusing: what's "next"? What's "previous"? Why's there no "current"?
   All it does is constructing a `PageRequest` from the two pagination-parameters.
   
   Why not call this function `constructFromParameters`?



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import jakarta.annotation.Nullable;
+
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageRequest {
+  private final @Nullable String innerPageToken;
+  private final int pageSize;
+
+  PageRequest(@Nullable String innerPageToken, int pageSize) {
+    this.innerPageToken = innerPageToken;
+    this.pageSize = pageSize;
+  }
+
+  /** Represents a non-paginated request. */
+  public static PageRequest readEverything() {

Review Comment:
   Maybe rename to `notPaged()`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2144157471


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;

Review Comment:
   The current approach does allow for future extensions. Different "data pointers" can be introduced when a Persistence impl. needs them. Since the "data pointer" always makes a round trip from Persistence code to the client and back to the _same_ persistence code, that code is free to introduce new encoding/decoding methods without affecting anything else.
   
   It is remotely possible that a page token ends up in an API method call different from the one that produced it. However, having "data pointer" sub-types (as opposed to just pairs of encode/decode methods) does not add any safety here. We could encode some "place" identifier into the token to bind it firmly to the API method, but I think it's an overkill.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140790271


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item

Review Comment:
   The java type `PageToken` is a container for a tuple of:
   1. Page size
   2. the "data pointer"
   
   The latter is an opaque piece of data as far as the java type `PageToken` is concerned. "Data pointers" are interpreted only by the code that actually produces them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140786069


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java:
##########
@@ -361,13 +362,19 @@ public List<EntityNameLookupRecord> lookupEntityActiveBatchInCurrentTxn(
             .map(
                 nameRecord ->
                     this.lookupEntityInCurrentTxn(
-                        callCtx, catalogId, nameRecord.getId(), entityType.getCode()))
-            .filter(entityFilter);
-    if (pageToken instanceof HasPageSize) {
-      data = data.limit(((HasPageSize) pageToken).getPageSize());
+                        callCtx, catalogId, nameRecord.getId(), entityType.getCode()));
+
+    Predicate<PolarisBaseEntity> tokenFilter = e -> true;
+    if (pageToken.hasDataReference()) {

Review Comment:
   refactored as suggested :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139135098


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,82 +18,85 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import static com.google.common.base.Preconditions.checkState;
 
-/**
- * Represents a page token that can be used by operations like `listTables`. Clients that specify a
- * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
- * which is a serialized PageToken.
- *
- * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
- * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
- * response, that means there is no more data to read.
- */
-public abstract class PageToken {
+import jakarta.annotation.Nullable;
 
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageToken {
+  private final @Nullable String encodedDataReference;
+  private final int pageSize;
 
-  /** Build a new PageToken from an input String, without a specified page size */
-  public static PageToken fromString(String token) {
-    return build(token, null);
+  PageToken(@Nullable String encodedDataReference, int pageSize) {
+    this.encodedDataReference = encodedDataReference;
+    this.pageSize = pageSize;
   }
 
-  /** Build a new PageToken from a limit */
-  public static PageToken fromLimit(Integer pageSize) {
-    return build(null, pageSize);
+  /** Represents a non-paginated request. */
+  public static PageToken readEverything() {
+    return new PageToken(null, -1);
   }
 
-  /** Build a {@link PageToken} from the input string and page size */
-  public static PageToken build(String token, Integer pageSize) {
-    if (token == null || token.isEmpty()) {
-      if (pageSize != null) {
-        return new LimitPageToken(pageSize);
-      } else {
-        return new ReadEverythingPageToken();
-      }
-    } else {
-      // TODO implement, split out by the token's prefix
-      throw new IllegalArgumentException("Unrecognized page token: " + token);
-    }
+  /** Represents a request to start paginating with a particular page size. */
+  public static PageToken fromLimit(int limit) {
+    return new PageToken(null, limit);
   }
 
-  /** Serialize a {@link PageToken} into a string */
-  public abstract String toTokenString();
+  /**
+   * Reconstructs a page token from the API-level page token string (returned to the client in the
+   * response to a previous request for similar data) and an API-level new requested page size.

Review Comment:
   What does "API-level" mean here? When we get a table identifier in a method like `IcebergCatalog.newTableOps`, we don't say it's an API-level TableIdentifier just because it came in via the API.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139136585


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPaging.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+/** Utility class for processing pagination of streams of entities ordered by some ID property. */
+public final class EntityIdPaging {
+  private EntityIdPaging() {}
+
+  /**
+   * Produces the reference to the next page of data in the form of a numerical entity ID. Entities
+   * in the associated stream of data are assumed to be ordered by ID.

Review Comment:
   What stream? This is just taking one entity and returning `Long.toString(entity.getId())`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140582261


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import jakarta.annotation.Nullable;
+
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageRequest {
+  private final @Nullable String innerPageToken;
+  private final int pageSize;
+
+  PageRequest(@Nullable String innerPageToken, int pageSize) {
+    this.innerPageToken = innerPageToken;
+    this.pageSize = pageSize;
+  }
+
+  /** Represents a non-paginated request. */
+  public static PageRequest readEverything() {
+    return new PageRequest(null, -1);
+  }
+
+  /** Represents a request to start paginating with a particular page size. */
+  public static PageRequest firstPage(int limit) {
+    return new PageRequest(null, limit);
+  }
+
+  /**
+   * Reconstructs a page request from the previous API-level page token and an API-level new
+   * requested page size.
+   *
+   * @param previousPageToken page token from the {@link Page#encodedResponseToken() previous page}
+   * @param requestedPageSize optional page size for the next page. If not set, the page size of the
+   *     previus page (encoded in the previous page token) will be reused.
+   */
+  public static PageRequest nextPage(

Review Comment:
   refactored to `PageToken.decode()`... Is it better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140599462


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+public class EntityIdPageToken implements PageToken {
+  private final boolean hasMore;
+  private final boolean hasPrevious;
+  private final long entityId;
+
+  private EntityIdPageToken(boolean hasMore, boolean hasPrevious, long entityId) {
+    this.hasMore = hasMore;
+    this.hasPrevious = hasPrevious;
+    this.entityId = entityId;
+  }
+
+  public static EntityIdPageToken forPage(List<PolarisBaseEntity> data) {
+    if (data.isEmpty()) {
+      return forLastItem(null);
+    }
+
+    return forLastItem(data.get(data.size() - 1));
+  }
+
+  public static EntityIdPageToken forLastItem(PolarisBaseEntity entity) {

Review Comment:
   reimplemented, please review again :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140604466


##########
persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkStore.java:
##########
@@ -294,14 +295,25 @@ List<ModelEntity> lookupFullEntitiesActive(
 
     // Currently check against ENTITIES not joining with ENTITIES_ACTIVE
     String hql =
-        "SELECT m from ModelEntity m where m.catalogId=:catalogId and m.parentId=:parentId and m.typeCode=:typeCode";
+        "SELECT m from ModelEntity m where"
+            + " m.catalogId=:catalogId and m.parentId=:parentId and m.typeCode=:typeCode and m.id > :tokenId";
+
+    long tokenId = -1;
+    if (pageToken.paginationRequested()) {
+      hql += " order by m.id asc";
+
+      if (pageToken.hasDataReference()) {

Review Comment:
   Polymorphism at the java type level is not necessary in this case. Users of the "data pointer" know how to parse it based on the API method called. It is not a matter of providing a sub-type, but about understanding the format.
   
   Currently parsing is achieved via static methods in `EntityIdPaging`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140576716


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.util.Optional;
+
+public final class PageTokenUtil {

Review Comment:
   done



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,84 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
  * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
  * continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageRequest request;
+  private final PageToken nextToken;
+  private final List<T> items;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageRequest request, PageToken nextToken, List<T> items) {
+    this.request = request;
+    this.nextToken = nextToken;
     this.items = items;
   }
 
-  /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
-   */
-  public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+  public static <T> Page<T> allItems(List<T> items) {
+    return new Page<>(PageRequest.readEverything(), () -> null, items);
+  }
+
+  public static <T> Page<T> fromStream(

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140595925


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -30,70 +29,8 @@
  * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
  * response, that means there is no more data to read.
  */
-public abstract class PageToken {
-
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
-
-  /** Build a new PageToken from an input String, without a specified page size */
-  public static PageToken fromString(String token) {
-    return build(token, null);
-  }
-
-  /** Build a new PageToken from a limit */
-  public static PageToken fromLimit(Integer pageSize) {
-    return build(null, pageSize);
-  }
-
-  /** Build a {@link PageToken} from the input string and page size */
-  public static PageToken build(String token, Integer pageSize) {
-    if (token == null || token.isEmpty()) {
-      if (pageSize != null) {
-        return new LimitPageToken(pageSize);
-      } else {
-        return new ReadEverythingPageToken();
-      }
-    } else {
-      // TODO implement, split out by the token's prefix
-      throw new IllegalArgumentException("Unrecognized page token: " + token);
-    }
-  }
-
+public abstract interface PageToken {

Review Comment:
   reimplemented.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140597771


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,8 +18,7 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import jakarta.annotation.Nullable;
 
 /**
  * Represents a page token that can be used by operations like `listTables`. Clients that specify a

Review Comment:
   reimplemented



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139129762


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item

Review Comment:
   I'm not sure what this is or why we're calling it a "data pointer". it looks like a function that converts a list of elements into a page token, except for some reason we're not using the Java type PageToken and we're going directly to the serialized representation of a PageToken



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139133630


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;

Review Comment:
   If a token has multiple components, should that be modeled as one "encoded data reference"? Multiple? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2145662600


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPaging.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+/** Utility class for processing pagination of streams of entities ordered by some ID property. */
+public final class EntityIdPaging {
+  private EntityIdPaging() {}
+
+  /**
+   * Produces the reference to the next page of data in the form of a numerical entity ID. Entities
+   * in the associated stream of data are assumed to be ordered by ID.

Review Comment:
   I found this a little confusing, as the doc doesn't seem to match the actual behavior of the method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139122216


##########
persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkStore.java:
##########
@@ -294,14 +295,25 @@ List<ModelEntity> lookupFullEntitiesActive(
 
     // Currently check against ENTITIES not joining with ENTITIES_ACTIVE
     String hql =
-        "SELECT m from ModelEntity m where m.catalogId=:catalogId and m.parentId=:parentId and m.typeCode=:typeCode";
+        "SELECT m from ModelEntity m where"
+            + " m.catalogId=:catalogId and m.parentId=:parentId and m.typeCode=:typeCode and m.id > :tokenId";
+
+    long tokenId = -1;

Review Comment:
   There used to be a constant for this in `EntityIdPageToken`; putting a magic number here is not a good idea



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139128145


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java:
##########
@@ -361,13 +362,19 @@ public List<EntityNameLookupRecord> lookupEntityActiveBatchInCurrentTxn(
             .map(
                 nameRecord ->
                     this.lookupEntityInCurrentTxn(
-                        callCtx, catalogId, nameRecord.getId(), entityType.getCode()))
-            .filter(entityFilter);
-    if (pageToken instanceof HasPageSize) {
-      data = data.limit(((HasPageSize) pageToken).getPageSize());
+                        callCtx, catalogId, nameRecord.getId(), entityType.getCode()));
+
+    Predicate<PolarisBaseEntity> tokenFilter = e -> true;
+    if (pageToken.hasDataReference()) {

Review Comment:
   In cases like this, it's better to do a single assignment to prevent anything from breaking if lines get moved around:
   ```
   final Predicate<PolarisBaseEntity> tokenFilter;
   if (pageToken.hasDataReference()) {
     tokenFilter = e -> e.getId() > nextId;
   } else {
     tokenFilter = e -> true;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139124237


##########
polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java:
##########
@@ -217,6 +217,7 @@ public static void enforceFeatureEnabledOrThrow(
   public static final PolarisConfiguration<Boolean> LIST_PAGINATION_ENABLED =
       PolarisConfiguration.<Boolean>builder()
           .key("LIST_PAGINATION_ENABLED")
+          .catalogConfig("polaris.config.list-pagination-enabled")

Review Comment:
   Unless you change the check in `IcebergCatalog`, you're just adding a flag that does nothing here



##########
polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java:
##########
@@ -217,6 +217,7 @@ public static void enforceFeatureEnabledOrThrow(
   public static final PolarisConfiguration<Boolean> LIST_PAGINATION_ENABLED =
       PolarisConfiguration.<Boolean>builder()
           .key("LIST_PAGINATION_ENABLED")
+          .catalogConfig("polaris.config.list-pagination-enabled")

Review Comment:
   Unless you change the check in `IcebergCatalog`, you're adding a flag that does nothing here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139133830


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;
+    if (requestedPageToken != null) {
+      byte[] bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      String token = new String(bytes, StandardCharsets.UTF_8);
+      int idx = token.indexOf(':');
+      checkArgument(idx > 0, "Invalid page token: %s", requestedPageToken);
+      int prevPageSize = Integer.parseInt(token.substring(0, idx));
+      encodedDataReference = token.substring(idx + 1);
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(prevPageSize);
+    } else {
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(-1);
+    }
+
+    return new PageToken(encodedDataReference, pageSize);
+  }
+
+  static @Nullable String encodePageToken(

Review Comment:
   Why is this not just a member of `PageToken`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139131910


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,82 +18,85 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import static com.google.common.base.Preconditions.checkState;
 
-/**
- * Represents a page token that can be used by operations like `listTables`. Clients that specify a
- * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
- * which is a serialized PageToken.
- *
- * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
- * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
- * response, that means there is no more data to read.
- */
-public abstract class PageToken {
+import jakarta.annotation.Nullable;
 
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
+/** A wrapper for pagination information passed in as part of a request. */

Review Comment:
   Why only a request? Isn't the response field called `next-page-token`? Is that not a PageToken?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2143215921


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item

Review Comment:
   I think we need a Java type for what you're calling a "data pointer" -- for what the API calls `page-token`. In #1555, I called this PageToken. If we need to wrap this PageToken in a PageRequest, that's okay.
   
   ```
   public record PageToken() {
     public void getEncodedString() { ... }
     public static PageToken fromEncodedString() { ... }
   }
   
   public record PageRequest(Optional<PageToken> pageToken, Optional<Integer> pageSize) {}
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140636443


##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -2538,7 +2518,7 @@ private PageToken buildPageToken(@Nullable String tokenString, @Nullable Integer
     if (!paginationEnabled) {
       return PageToken.readEverything();
     } else {
-      return PageToken.build(tokenString, pageSize);
+      return PageToken.decode(tokenString, pageSize);

Review Comment:
   renamed back to `.build()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139133162


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;
+    if (requestedPageToken != null) {

Review Comment:
   Looks like you've changed the proposed format of the tokens here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140593109


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import jakarta.annotation.Nullable;
+
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageRequest {
+  private final @Nullable String innerPageToken;
+  private final int pageSize;
+
+  PageRequest(@Nullable String innerPageToken, int pageSize) {
+    this.innerPageToken = innerPageToken;
+    this.pageSize = pageSize;
+  }
+
+  /** Represents a non-paginated request. */
+  public static PageRequest readEverything() {

Review Comment:
   @eric-maynard : Are you ok with this renaming?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140775294


##########
polaris-core/src/testFixtures/java/org/apache/polaris/core/persistence/PolarisTestMetaStoreManager.java:
##########
@@ -791,7 +793,8 @@ void dropEntity(List<PolarisEntityCore> catalogPath, PolarisBaseEntity entityToD
                       PolarisEntityType.CATALOG_ROLE,
                       PolarisEntitySubType.ANY_SUBTYPE,
                       PageToken.readEverything())
-                  .getEntities();
+                  .getEntities()
+                  .items();

Review Comment:
   restored `List` return type of `getEntities()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139121940


##########
persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkStore.java:
##########
@@ -294,14 +295,25 @@ List<ModelEntity> lookupFullEntitiesActive(
 
     // Currently check against ENTITIES not joining with ENTITIES_ACTIVE
     String hql =
-        "SELECT m from ModelEntity m where m.catalogId=:catalogId and m.parentId=:parentId and m.typeCode=:typeCode";
+        "SELECT m from ModelEntity m where"
+            + " m.catalogId=:catalogId and m.parentId=:parentId and m.typeCode=:typeCode and m.id > :tokenId";
+
+    long tokenId = -1;
+    if (pageToken.paginationRequested()) {
+      hql += " order by m.id asc";
+
+      if (pageToken.hasDataReference()) {

Review Comment:
   This is still polymorphism, we're just not taking advantage of the fact that this is built into the language already:
   ```
   if (pageToken isinstanceof EntityIdPageToken e) {
     tokenId = e.getEntityId();
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139124036


##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -2538,7 +2518,7 @@ private PageToken buildPageToken(@Nullable String tokenString, @Nullable Integer
     if (!paginationEnabled) {
       return PageToken.readEverything();
     } else {
-      return PageToken.build(tokenString, pageSize);
+      return PageToken.decode(tokenString, pageSize);

Review Comment:
   Is this really necessary? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140651796


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;

Review Comment:
   The interpretation of `encodedDataReference` is up to the code that constructs it for paged responses. Let's deal with complex cases when we have them in practice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140646691


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,82 +18,85 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import static com.google.common.base.Preconditions.checkState;
 
-/**
- * Represents a page token that can be used by operations like `listTables`. Clients that specify a
- * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
- * which is a serialized PageToken.
- *
- * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
- * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
- * response, that means there is no more data to read.
- */
-public abstract class PageToken {
+import jakarta.annotation.Nullable;
 
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
+/** A wrapper for pagination information passed in as part of a request. */

Review Comment:
   `next-page-token` at the API layer is not represented by a `PageToken` when constructing the response.
   
   Do you have a suggestion for a better javadoc here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140703611


##########
polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java:
##########
@@ -217,6 +217,7 @@ public static void enforceFeatureEnabledOrThrow(
   public static final PolarisConfiguration<Boolean> LIST_PAGINATION_ENABLED =
       PolarisConfiguration.<Boolean>builder()
           .key("LIST_PAGINATION_ENABLED")
+          .catalogConfig("polaris.config.list-pagination-enabled")

Review Comment:
   This was carried over from #1555. Did I miss some related changes?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2145656038


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;
+    if (requestedPageToken != null) {
+      byte[] bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      String token = new String(bytes, StandardCharsets.UTF_8);
+      int idx = token.indexOf(':');
+      checkArgument(idx > 0, "Invalid page token: %s", requestedPageToken);
+      int prevPageSize = Integer.parseInt(token.substring(0, idx));
+      encodedDataReference = token.substring(idx + 1);
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(prevPageSize);
+    } else {
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(-1);
+    }
+
+    return new PageToken(encodedDataReference, pageSize);
+  }
+
+  static @Nullable String encodePageToken(

Review Comment:
   I still don't understand why this encoding isn't an attribute of the `PageToken`. I think this may just be a side effect of us directly using the encoded string instead of a PageToken, as is discussed above.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2143218650


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;
+    if (requestedPageToken != null) {
+      byte[] bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      String token = new String(bytes, StandardCharsets.UTF_8);
+      int idx = token.indexOf(':');
+      checkArgument(idx > 0, "Invalid page token: %s", requestedPageToken);
+      int prevPageSize = Integer.parseInt(token.substring(0, idx));
+      encodedDataReference = token.substring(idx + 1);
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(prevPageSize);
+    } else {
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(-1);
+    }
+
+    return new PageToken(encodedDataReference, pageSize);
+  }
+
+  static @Nullable String encodePageToken(

Review Comment:
   You could co-locate them in PageToken, so I don't think that really answers the question



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2143217936


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;
+    if (requestedPageToken != null) {

Review Comment:
   It is; closing as duplicate of the other threads asking how this format can be extended



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Delineate pagination requests and tokens [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2138522313


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.util.Optional;
+
+public final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  public static PageRequest decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String token = null;
+    if (requestedPageToken != null) {
+      int idx = requestedPageToken.indexOf(':');

Review Comment:
   I'll keep the propagation of the previous page size into the next request via the page token. 
   
   I think doing that is reasonable to maintain better alignment with the REST API spec.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #1838:
URL: https://github.com/apache/polaris/pull/1838#issuecomment-2960662873

   Squashed, rebased, resolved conflicts and addressed some feedback. Please review again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139123008


##########
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java:
##########
@@ -459,29 +461,37 @@ public <T> Page<T> listEntities(
             entityType.getCode(),
             "realm_id",
             realmId);
+    Map<String, Object> whereGreater = Map.of();
 
     // Limit can't be pushed down, due to client side filtering
     // absence of transaction.
+    String orderByColumnName = null;
+    if (pageToken.paginationRequested()) {
+      orderByColumnName = ModelEntity.ID_COLUMN;
+      if (pageToken.hasDataReference()) {
+        long boundary = entityIdBoundary(pageToken);
+        whereGreater = Map.of(ModelEntity.ID_COLUMN, boundary);
+      }
+    }
+
     try {
       PreparedQuery query =
           QueryGenerator.generateSelectQuery(
-              ModelEntity.ALL_COLUMNS, ModelEntity.TABLE_NAME, params);
-      List<PolarisBaseEntity> results = new ArrayList<>();
+              ModelEntity.ALL_COLUMNS,
+              ModelEntity.TABLE_NAME,
+              whereEquals,
+              whereGreater,
+              orderByColumnName);
+      AtomicReference<Page<T>> results = new AtomicReference<>();
       datasourceOperations.executeSelectOverStream(
           query,
           new ModelEntity(),
           stream -> {
             var data = stream.filter(entityFilter);
-            if (pageToken instanceof HasPageSize hasPageSize) {
-              data = data.limit(hasPageSize.getPageSize());
-            }
-            data.forEach(results::add);
+            results.set(
+                Page.mapped(pageToken, data, transformer, EntityIdPaging::encodedDataReference));

Review Comment:
   Why are we pushing the limit into `Page.mapped`? This feels beyond the scope of what a `Page` should be handling. If we _are_ pushing it down, why not also push the filter?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139125723


##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -775,14 +771,10 @@ public List<Namespace> listNamespaces() {
 
   @Override
   public List<Namespace> listNamespaces(Namespace namespace) throws NoSuchNamespaceException {
-    return listNamespaces(namespace, PageToken.readEverything()).items;
-  }
-
-  public Page<Namespace> listNamespaces(Namespace namespace, String pageToken, Integer pageSize) {
-    return listNamespaces(namespace, buildPageToken(pageToken, pageSize));
+    return listNamespaces(namespace, PageToken.readEverything()).items();
   }
 
-  private Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)
+  public Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)

Review Comment:
   @VisibleForTesting?



##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -775,14 +771,10 @@ public List<Namespace> listNamespaces() {
 
   @Override
   public List<Namespace> listNamespaces(Namespace namespace) throws NoSuchNamespaceException {
-    return listNamespaces(namespace, PageToken.readEverything()).items;
-  }
-
-  public Page<Namespace> listNamespaces(Namespace namespace, String pageToken, Integer pageSize) {
-    return listNamespaces(namespace, buildPageToken(pageToken, pageSize));
+    return listNamespaces(namespace, PageToken.readEverything()).items();
   }
 
-  private Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)
+  public Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)

Review Comment:
   `@VisibleForTesting`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140628324


##########
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java:
##########
@@ -459,29 +461,37 @@ public <T> Page<T> listEntities(
             entityType.getCode(),
             "realm_id",
             realmId);
+    Map<String, Object> whereGreater = Map.of();
 
     // Limit can't be pushed down, due to client side filtering
     // absence of transaction.
+    String orderByColumnName = null;
+    if (pageToken.paginationRequested()) {
+      orderByColumnName = ModelEntity.ID_COLUMN;
+      if (pageToken.hasDataReference()) {
+        long boundary = entityIdBoundary(pageToken);
+        whereGreater = Map.of(ModelEntity.ID_COLUMN, boundary);
+      }
+    }
+
     try {
       PreparedQuery query =
           QueryGenerator.generateSelectQuery(
-              ModelEntity.ALL_COLUMNS, ModelEntity.TABLE_NAME, params);
-      List<PolarisBaseEntity> results = new ArrayList<>();
+              ModelEntity.ALL_COLUMNS,
+              ModelEntity.TABLE_NAME,
+              whereEquals,
+              whereGreater,
+              orderByColumnName);
+      AtomicReference<Page<T>> results = new AtomicReference<>();
       datasourceOperations.executeSelectOverStream(
           query,
           new ModelEntity(),
           stream -> {
             var data = stream.filter(entityFilter);
-            if (pageToken instanceof HasPageSize hasPageSize) {
-              data = data.limit(hasPageSize.getPageSize());
-            }
-            data.forEach(results::add);
+            results.set(
+                Page.mapped(pageToken, data, transformer, EntityIdPaging::encodedDataReference));

Review Comment:
   The filter (line 490) is implicitly part of the page construction login by being a stage in the `data` stream.
   
   The use of `PageToken.pageSize()` inside `Page.mapped()` is merely a convenience / guardrail to avoid assuming that the stream is limited to the exact page size.



##########
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java:
##########
@@ -459,29 +461,37 @@ public <T> Page<T> listEntities(
             entityType.getCode(),
             "realm_id",
             realmId);
+    Map<String, Object> whereGreater = Map.of();
 
     // Limit can't be pushed down, due to client side filtering
     // absence of transaction.
+    String orderByColumnName = null;
+    if (pageToken.paginationRequested()) {
+      orderByColumnName = ModelEntity.ID_COLUMN;
+      if (pageToken.hasDataReference()) {
+        long boundary = entityIdBoundary(pageToken);
+        whereGreater = Map.of(ModelEntity.ID_COLUMN, boundary);
+      }
+    }
+
     try {
       PreparedQuery query =
           QueryGenerator.generateSelectQuery(
-              ModelEntity.ALL_COLUMNS, ModelEntity.TABLE_NAME, params);
-      List<PolarisBaseEntity> results = new ArrayList<>();
+              ModelEntity.ALL_COLUMNS,
+              ModelEntity.TABLE_NAME,
+              whereEquals,
+              whereGreater,
+              orderByColumnName);
+      AtomicReference<Page<T>> results = new AtomicReference<>();
       datasourceOperations.executeSelectOverStream(
           query,
           new ModelEntity(),
           stream -> {
             var data = stream.filter(entityFilter);
-            if (pageToken instanceof HasPageSize hasPageSize) {
-              data = data.limit(hasPageSize.getPageSize());
-            }
-            data.forEach(results::add);
+            results.set(
+                Page.mapped(pageToken, data, transformer, EntityIdPaging::encodedDataReference));

Review Comment:
   The filter (line 490) is implicitly part of the page construction logic by being a stage in the `data` stream.
   
   The use of `PageToken.pageSize()` inside `Page.mapped()` is merely a convenience / guardrail to avoid assuming that the stream is limited to the exact page size.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on PR #1838:
URL: https://github.com/apache/polaris/pull/1838#issuecomment-3104093463

   IIUC #1938 supersedes this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2143228096


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item
+   *     from the previous page. The output of this function will be available from {@link
+   *     PageToken#encodedDataReference()} associated with the request for the next page.
+   */
+  public static <R, T> Page<R> mapped(
+      PageToken request, Stream<T> items, Function<T, R> mapper, Function<T, String> dataPointer) {
+    List<R> data;
+    if (request.paginationRequested()) {
+      data = new ArrayList<>(request.pageSize());
+    } else {
+      data = new ArrayList<>();
+    }
+
+    T last = null;
+    Iterator<T> it = iterator(items.spliterator());
+    while (it.hasNext() && (!request.paginationRequested() || data.size() < request.pageSize())) {
+      last = it.next();
+      data.add(mapper.apply(last));
+    }
+
+    if (request.paginationRequested() && data.size() < request.pageSize()) {
+      // the page was not filled, inform the client that there's no next page
+      last = null;
+    }
+
+    return new Page<>(request, dataPointer.apply(last), data);
+  }
+
+  public List<T> items() {
+    return items;
+  }
+
+  /**
+   * Returns a page token in encoded form suitable for returning to API clients. The string returned
+   * from this method is expected to be parsed by {@link PageToken#decode(String, Integer)} when
+   * servicing the request for the next page of related data.
+   */
+  public @Nullable String encodedResponseToken() {
+    return PageTokenUtil.encodePageToken(request, encodedDataReference);

Review Comment:
   Yeah I think it would be reasonable for callers to just call PageTokenUtil.encodePageToken directly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2143211070


##########
polaris-core/src/main/java/org/apache/polaris/core/catalog/PolarisCatalogHelpers.java:
##########
@@ -62,20 +61,28 @@ public static Namespace getParentNamespace(Namespace namespace) {
     return Namespace.of(parentLevels);
   }
 
-  public static List<Namespace> nameAndIdToNamespaces(
-      List<PolarisEntity> catalogPath, List<PolarisEntity.NameAndId> entities) {
+  public static Namespace nameAndIdToNamespace(

Review Comment:
   What I mean is that I'm not clear on why this change is needed to implement pagination at the persistence level



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2144145119


##########
polaris-core/src/main/java/org/apache/polaris/core/catalog/PolarisCatalogHelpers.java:
##########
@@ -62,20 +61,28 @@ public static Namespace getParentNamespace(Namespace namespace) {
     return Namespace.of(parentLevels);
   }
 
-  public static List<Namespace> nameAndIdToNamespaces(
-      List<PolarisEntity> catalogPath, List<PolarisEntity.NameAndId> entities) {
+  public static Namespace nameAndIdToNamespace(

Review Comment:
   This is a side effect of introducing the `Page.map()` method. Pages are produced by persistence and hold the low-level "data pointer". The new `map()` method allows transforming the items in a `Page` while maintaining the "data pointer" (and not having to "manually" pass it around).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2144161693


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item
+   *     from the previous page. The output of this function will be available from {@link
+   *     PageToken#encodedDataReference()} associated with the request for the next page.
+   */
+  public static <R, T> Page<R> mapped(
+      PageToken request, Stream<T> items, Function<T, R> mapper, Function<T, String> dataPointer) {
+    List<R> data;
+    if (request.paginationRequested()) {
+      data = new ArrayList<>(request.pageSize());
+    } else {
+      data = new ArrayList<>();
+    }
+
+    T last = null;
+    Iterator<T> it = iterator(items.spliterator());
+    while (it.hasNext() && (!request.paginationRequested() || data.size() < request.pageSize())) {
+      last = it.next();
+      data.add(mapper.apply(last));
+    }
+
+    if (request.paginationRequested() && data.size() < request.pageSize()) {
+      // the page was not filled, inform the client that there's no next page
+      last = null;
+    }
+
+    return new Page<>(request, dataPointer.apply(last), data);
+  }
+
+  public List<T> items() {
+    return items;
+  }
+
+  /**
+   * Returns a page token in encoded form suitable for returning to API clients. The string returned
+   * from this method is expected to be parsed by {@link PageToken#decode(String, Integer)} when
+   * servicing the request for the next page of related data.
+   */
+  public @Nullable String encodedResponseToken() {
+    return PageTokenUtil.encodePageToken(request, encodedDataReference);

Review Comment:
   I can make this change if you prefer. No worries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139126077


##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -798,13 +790,12 @@ private Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)
                 PolarisEntityType.NAMESPACE,
                 PolarisEntitySubType.NULL_SUBTYPE,
                 pageToken);
-    List<PolarisEntity.NameAndId> entities =
-        PolarisEntity.toNameAndIdList(listResult.getEntities());
-    List<Namespace> namespaces = PolarisCatalogHelpers.nameAndIdToNamespaces(catalogPath, entities);
     return listResult
-        .getPageToken()
-        .map(token -> new Page<>(token, namespaces))
-        .orElseGet(() -> Page.fromItems(namespaces));
+        .getEntities()
+        .map(
+            record ->

Review Comment:
   Why `record`?



##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -798,13 +790,12 @@ private Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)
                 PolarisEntityType.NAMESPACE,
                 PolarisEntitySubType.NULL_SUBTYPE,
                 pageToken);
-    List<PolarisEntity.NameAndId> entities =
-        PolarisEntity.toNameAndIdList(listResult.getEntities());
-    List<Namespace> namespaces = PolarisCatalogHelpers.nameAndIdToNamespaces(catalogPath, entities);
     return listResult
-        .getPageToken()
-        .map(token -> new Page<>(token, namespaces))
-        .orElseGet(() -> Page.fromItems(namespaces));
+        .getEntities()
+        .map(
+            record ->

Review Comment:
   On the whole, this doesn't look less complicated than the existing setup to me



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139128861


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item
+   *     from the previous page. The output of this function will be available from {@link
+   *     PageToken#encodedDataReference()} associated with the request for the next page.
+   */
+  public static <R, T> Page<R> mapped(
+      PageToken request, Stream<T> items, Function<T, R> mapper, Function<T, String> dataPointer) {
+    List<R> data;
+    if (request.paginationRequested()) {
+      data = new ArrayList<>(request.pageSize());
+    } else {
+      data = new ArrayList<>();
+    }
+
+    T last = null;
+    Iterator<T> it = iterator(items.spliterator());
+    while (it.hasNext() && (!request.paginationRequested() || data.size() < request.pageSize())) {
+      last = it.next();
+      data.add(mapper.apply(last));
+    }
+
+    if (request.paginationRequested() && data.size() < request.pageSize()) {
+      // the page was not filled, inform the client that there's no next page
+      last = null;
+    }
+
+    return new Page<>(request, dataPointer.apply(last), data);
+  }
+
+  public List<T> items() {
+    return items;
+  }
+
+  /**
+   * Returns a page token in encoded form suitable for returning to API clients. The string returned
+   * from this method is expected to be parsed by {@link PageToken#decode(String, Integer)} when
+   * servicing the request for the next page of related data.
+   */
+  public @Nullable String encodedResponseToken() {

Review Comment:
   Why is turning a `PageToken` into a string the responsibility of `Page` and not `PageToken`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139137029


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPaging.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+/** Utility class for processing pagination of streams of entities ordered by some ID property. */
+public final class EntityIdPaging {

Review Comment:
   I'm not sure how useful this is going to be, since its methods are `static` there can't be another implementation. The methods both look very thin.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2139132668


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,82 +18,85 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import static com.google.common.base.Preconditions.checkState;
 
-/**
- * Represents a page token that can be used by operations like `listTables`. Clients that specify a
- * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
- * which is a serialized PageToken.
- *
- * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
- * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
- * response, that means there is no more data to read.
- */
-public abstract class PageToken {
+import jakarta.annotation.Nullable;
 
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageToken {
+  private final @Nullable String encodedDataReference;

Review Comment:
   What's a "data reference"? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140641544


##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -775,14 +771,10 @@ public List<Namespace> listNamespaces() {
 
   @Override
   public List<Namespace> listNamespaces(Namespace namespace) throws NoSuchNamespaceException {
-    return listNamespaces(namespace, PageToken.readEverything()).items;
-  }
-
-  public Page<Namespace> listNamespaces(Namespace namespace, String pageToken, Integer pageSize) {
-    return listNamespaces(namespace, buildPageToken(pageToken, pageSize));
+    return listNamespaces(namespace, PageToken.readEverything()).items();
   }
 
-  private Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)
+  public Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)

Review Comment:
   It has non-test callers instead of the removed `public Page<Namespace> listNamespaces(Namespace namespace, String pageToken, Integer pageSize)` method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140630622


##########
polaris-core/src/main/java/org/apache/polaris/core/catalog/PolarisCatalogHelpers.java:
##########
@@ -62,20 +61,28 @@ public static Namespace getParentNamespace(Namespace namespace) {
     return Namespace.of(parentLevels);
   }
 
-  public static List<Namespace> nameAndIdToNamespaces(
-      List<PolarisEntity> catalogPath, List<PolarisEntity.NameAndId> entities) {
+  public static Namespace nameAndIdToNamespace(

Review Comment:
   WDYM "why"? It is called from `IcebergCatalog`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140663673


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPaging.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+/** Utility class for processing pagination of streams of entities ordered by some ID property. */
+public final class EntityIdPaging {

Review Comment:
   `EntityIdPaging` is a specific case of constructing and parsing `PageToken.encodedDataReference()` for Persistence implementations that rely of sorted streams of entities.
   
   Having these `static` methods allows having the encoding/decoding code across several use cases in the current Polaris code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140648325


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,82 +18,85 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import static com.google.common.base.Preconditions.checkState;
 
-/**
- * Represents a page token that can be used by operations like `listTables`. Clients that specify a
- * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
- * which is a serialized PageToken.
- *
- * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
- * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
- * response, that means there is no more data to read.
- */
-public abstract class PageToken {
+import jakarta.annotation.Nullable;
 
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
+/** A wrapper for pagination information passed in as part of a request. */
+public class PageToken {
+  private final @Nullable String encodedDataReference;

Review Comment:
   Explained by the javadoc on `encodedDataReference()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140779943


##########
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -798,13 +790,12 @@ private Page<Namespace> listNamespaces(Namespace namespace, PageToken pageToken)
                 PolarisEntityType.NAMESPACE,
                 PolarisEntitySubType.NULL_SUBTYPE,
                 pageToken);
-    List<PolarisEntity.NameAndId> entities =
-        PolarisEntity.toNameAndIdList(listResult.getEntities());
-    List<Namespace> namespaces = PolarisCatalogHelpers.nameAndIdToNamespaces(catalogPath, entities);
     return listResult
-        .getPageToken()
-        .map(token -> new Page<>(token, namespaces))
-        .orElseGet(() -> Page.fromItems(namespaces));
+        .getEntities()
+        .map(
+            record ->

Review Comment:
   This change is not about simplification, but about controlling the propagation of the page token inside the `Page` class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2143217174


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java:
##########
@@ -18,82 +18,85 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
-import java.util.List;
-import java.util.Objects;
+import static com.google.common.base.Preconditions.checkState;
 
-/**
- * Represents a page token that can be used by operations like `listTables`. Clients that specify a
- * `pageSize` (or a `pageToken`) may receive a `next-page-token` in the response, the content of
- * which is a serialized PageToken.
- *
- * <p>By providing that in the next query's `pageToken`, the client can resume listing where they
- * left off. If the client provides a `pageToken` or `pageSize` but `next-page-token` is null in the
- * response, that means there is no more data to read.
- */
-public abstract class PageToken {
+import jakarta.annotation.Nullable;
 
-  /** Build a new PageToken that reads everything */
-  public static PageToken readEverything() {
-    return build(null, null);
-  }
+/** A wrapper for pagination information passed in as part of a request. */

Review Comment:
   I think this is more of less a duplicate of [this other thread](https://github.com/apache/polaris/pull/1838#discussion_r2143215921) -- for me, it's not clear why PageToken is only used on one side here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard closed pull request #1838: Persistence implementation for pagination in some requests 
URL: https://github.com/apache/polaris/pull/1838


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1838:
URL: https://github.com/apache/polaris/pull/1838#issuecomment-3067554015

   This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140699607


##########
persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkStore.java:
##########
@@ -294,14 +295,25 @@ List<ModelEntity> lookupFullEntitiesActive(
 
     // Currently check against ENTITIES not joining with ENTITIES_ACTIVE
     String hql =
-        "SELECT m from ModelEntity m where m.catalogId=:catalogId and m.parentId=:parentId and m.typeCode=:typeCode";
+        "SELECT m from ModelEntity m where"
+            + " m.catalogId=:catalogId and m.parentId=:parentId and m.typeCode=:typeCode and m.id > :tokenId";
+
+    long tokenId = -1;

Review Comment:
   refactored



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140659436


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPaging.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+/** Utility class for processing pagination of streams of entities ordered by some ID property. */
+public final class EntityIdPaging {
+  private EntityIdPaging() {}
+
+  /**
+   * Produces the reference to the next page of data in the form of a numerical entity ID. Entities
+   * in the associated stream of data are assumed to be ordered by ID.

Review Comment:
   This is in reference to how the document method is used in relation to handling paginated requests and responses (which deal with streams of data).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Delineate pagination requests and tokens [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2138362942


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.util.Optional;
+
+public final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  public static PageRequest decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String token = null;
+    if (requestedPageToken != null) {
+      int idx = requestedPageToken.indexOf(':');

Review Comment:
   > AFAIK the page-size is included in every request
   That's not correct



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.util.Optional;
+
+public final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  public static PageRequest decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String token = null;
+    if (requestedPageToken != null) {
+      int idx = requestedPageToken.indexOf(':');

Review Comment:
   > AFAIK the page-size is included in every request
   
   That's not correct



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Delineate pagination requests and tokens [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2138362942


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.util.Optional;
+
+public final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  public static PageRequest decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String token = null;
+    if (requestedPageToken != null) {
+      int idx = requestedPageToken.indexOf(':');

Review Comment:
   > AFAIK the page-size is included in every request
   
   That's not correct per the spec



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140599925


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+public class EntityIdPageToken implements PageToken {
+  private final boolean hasMore;
+  private final boolean hasPrevious;
+  private final long entityId;
+
+  private EntityIdPageToken(boolean hasMore, boolean hasPrevious, long entityId) {
+    this.hasMore = hasMore;
+    this.hasPrevious = hasPrevious;
+    this.entityId = entityId;
+  }
+
+  public static EntityIdPageToken forPage(List<PolarisBaseEntity> data) {
+    if (data.isEmpty()) {
+      return forLastItem(null);
+    }
+
+    return forLastItem(data.get(data.size() - 1));
+  }
+
+  public static EntityIdPageToken forLastItem(PolarisBaseEntity entity) {
+    if (entity == null) {
+      return new EntityIdPageToken(false, false, 0);
+    }
+    return new EntityIdPageToken(true, true, entity.getId());
+  }
+
+  /**
+   * Extracts an {@link EntityIdPageToken} from a {@link PageRequest} assuming the request is a
+   * continuation of the API call that previously produced an {@link EntityIdPageToken}.
+   */
+  public static EntityIdPageToken fromPageRequest(PageRequest pageRequest) {
+    String token = pageRequest.innerPageToken();
+    if (token == null) {
+      return new EntityIdPageToken(false, false, 0);
+    } else {
+      return new EntityIdPageToken(false, true, Long.parseLong(token));
+    }
+  }
+
+  public long getId() {
+    return entityId;
+  }
+
+  public boolean hasPrevious() {

Review Comment:
   reimplemented, please review again :)



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import java.util.List;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+public class EntityIdPageToken implements PageToken {
+  private final boolean hasMore;

Review Comment:
   reimplemented, please review again :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140653705


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item
+   *     from the previous page. The output of this function will be available from {@link
+   *     PageToken#encodedDataReference()} associated with the request for the next page.
+   */
+  public static <R, T> Page<R> mapped(
+      PageToken request, Stream<T> items, Function<T, R> mapper, Function<T, String> dataPointer) {
+    List<R> data;
+    if (request.paginationRequested()) {
+      data = new ArrayList<>(request.pageSize());
+    } else {
+      data = new ArrayList<>();
+    }
+
+    T last = null;
+    Iterator<T> it = iterator(items.spliterator());
+    while (it.hasNext() && (!request.paginationRequested() || data.size() < request.pageSize())) {
+      last = it.next();
+      data.add(mapper.apply(last));
+    }
+
+    if (request.paginationRequested() && data.size() < request.pageSize()) {
+      // the page was not filled, inform the client that there's no next page
+      last = null;
+    }
+
+    return new Page<>(request, dataPointer.apply(last), data);
+  }
+
+  public List<T> items() {
+    return items;
+  }
+
+  /**
+   * Returns a page token in encoded form suitable for returning to API clients. The string returned
+   * from this method is expected to be parsed by {@link PageToken#decode(String, Integer)} when
+   * servicing the request for the next page of related data.
+   */
+  public @Nullable String encodedResponseToken() {
+    return PageTokenUtil.encodePageToken(request, encodedDataReference);

Review Comment:
   Yes, it does :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2140652910


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.api.client.util.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Optional;
+
+final class PageTokenUtil {
+
+  private PageTokenUtil() {}
+
+  static PageToken decodePageRequest(
+      @Nullable String requestedPageToken, @Nullable Integer requestedPageSize) {
+    int pageSize;
+    String encodedDataReference = null;
+    if (requestedPageToken != null) {
+      byte[] bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      String token = new String(bytes, StandardCharsets.UTF_8);
+      int idx = token.indexOf(':');
+      checkArgument(idx > 0, "Invalid page token: %s", requestedPageToken);
+      int prevPageSize = Integer.parseInt(token.substring(0, idx));
+      encodedDataReference = token.substring(idx + 1);
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(prevPageSize);
+    } else {
+      pageSize = Optional.ofNullable(requestedPageSize).orElse(-1);
+    }
+
+    return new PageToken(encodedDataReference, pageSize);
+  }
+
+  static @Nullable String encodePageToken(

Review Comment:
   To co-locate encoding and decoding code in one java file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2144152233


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item

Review Comment:
   How about keeping current `PageToken` class mostly as it is, but replacing its `String encodedDataReference` with `DataPointer dataPointer`?
   
   It does make the code nicer by having a specific type, but IMHO it adds a difficulty from the processing perspective:
   * Parsing `DataPointer` to a specific sub-type when `PageToken` is constructed will require encoding type information in the token. That is redundant IMHO, but I'm ok with that change.
   * Persistence methods that need to "understand" a `DataPointer` will have to be type casts, which is not elegant, IMHO. Currently they do not have to, persistence implementations currently just call a known parse method, which then will raise exceptions if the data is malformed.
   
   As a middle ground option WDYT about using `DataPointer` to wrap the encoded `String` (no sub-types), but still use static parsing methods to be called from Persistence?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2143232672


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPaging.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+/** Utility class for processing pagination of streams of entities ordered by some ID property. */
+public final class EntityIdPaging {

Review Comment:
   I see, but I'm trying to understand what this will look like if we have `SomeOtherEntityIdPaging`. Will callers have some way of selecting a `...Paging` based on the token structure or method?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2143212026


##########
polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java:
##########
@@ -217,6 +217,7 @@ public static void enforceFeatureEnabledOrThrow(
   public static final PolarisConfiguration<Boolean> LIST_PAGINATION_ENABLED =
       PolarisConfiguration.<Boolean>builder()
           .key("LIST_PAGINATION_ENABLED")
+          .catalogConfig("polaris.config.list-pagination-enabled")

Review Comment:
   If we add the catalog config here, we need to pass the catalog in when the flag is actually used to get a config value



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2144161141


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java:
##########
@@ -18,25 +18,96 @@
  */
 package org.apache.polaris.core.persistence.pagination;
 
+import static java.util.Spliterators.iterator;
+
+import jakarta.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Iterator;
 import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 /**
- * An immutable page of items plus their paging cursor. The {@link PageToken} here can be used to
- * continue the listing operation that generated the `items`.
+ * An immutable page of items plus their paging cursor. The {@link #encodedResponseToken()} here can
+ * be used to continue the listing operation that generated the `items`.
  */
 public class Page<T> {
-  public final PageToken pageToken;
-  public final List<T> items;
+  private final PageToken request;
+  private final List<T> items;
+  @Nullable private final String encodedDataReference;
 
-  public Page(PageToken pageToken, List<T> items) {
-    this.pageToken = pageToken;
+  private Page(PageToken request, @Nullable String encodedDataReference, List<T> items) {
+    this.request = request;
+    this.encodedDataReference = encodedDataReference;
     this.items = items;
   }
 
   /**
-   * Used to wrap a {@link List<T>} of items into a {@link Page <T>} when there are no more pages
+   * Builds a complete response page for the full list of relevant items. No subsequence pages of
+   * related data exist.
    */
   public static <T> Page<T> fromItems(List<T> items) {
-    return new Page<>(new DonePageToken(), items);
+    return new Page<>(PageToken.readEverything(), null, items);
+  }
+
+  /**
+   * Produces a response page by consuming the number of items from the provided stream according to
+   * the {@code request} parameter. Source items can be converted to a different type by providing a
+   * {@code mapper} function. The page token for the response will be produced from the request data
+   * combined with the pointer to the next page of data provided by the {@code dataPointer}
+   * function.
+   *
+   * @param request defines pagination parameters that were uses to produce this page of data.
+   * @param items stream of source data
+   * @param mapper converter from source data types to response data types.
+   * @param dataPointer determines the internal pointer to the next page of data given the last item
+   *     from the previous page. The output of this function will be available from {@link
+   *     PageToken#encodedDataReference()} associated with the request for the next page.
+   */
+  public static <R, T> Page<R> mapped(
+      PageToken request, Stream<T> items, Function<T, R> mapper, Function<T, String> dataPointer) {
+    List<R> data;
+    if (request.paginationRequested()) {
+      data = new ArrayList<>(request.pageSize());
+    } else {
+      data = new ArrayList<>();
+    }
+
+    T last = null;
+    Iterator<T> it = iterator(items.spliterator());
+    while (it.hasNext() && (!request.paginationRequested() || data.size() < request.pageSize())) {
+      last = it.next();
+      data.add(mapper.apply(last));
+    }
+
+    if (request.paginationRequested() && data.size() < request.pageSize()) {
+      // the page was not filled, inform the client that there's no next page
+      last = null;
+    }
+
+    return new Page<>(request, dataPointer.apply(last), data);
+  }
+
+  public List<T> items() {
+    return items;
+  }
+
+  /**
+   * Returns a page token in encoded form suitable for returning to API clients. The string returned
+   * from this method is expected to be parsed by {@link PageToken#decode(String, Integer)} when
+   * servicing the request for the next page of related data.
+   */
+  public @Nullable String encodedResponseToken() {
+    return PageTokenUtil.encodePageToken(request, encodedDataReference);

Review Comment:
   Calling `PageTokenUtil.encodePageToken(Page)` from a place that has a reference to a `Page` is less elegant than calling `Page.encodedResponseToken()`... This is a matter of opinion, of course :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Persistence implementation for pagination in some requests [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #1838:
URL: https://github.com/apache/polaris/pull/1838#discussion_r2144165847


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPaging.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import jakarta.annotation.Nullable;
+import org.apache.polaris.core.entity.PolarisBaseEntity;
+
+/** Utility class for processing pagination of streams of entities ordered by some ID property. */
+public final class EntityIdPaging {

Review Comment:
   If `SomeOtherEntityIdPaging` produces a single primitive value from a "data pointer" it would be very similar to `EntityIdPaging`, I guess.
   
   If it produces a rich value, then it will need a companion type to hold the parsed data... but that is something to be done if and when we need it. It could be something like `static NameAndIdPaging.entityBoundary(PageToken pageToken)` returning a new `NameAndIdPaging.Boundary(name, id)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org