You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "youngrae81 (via GitHub)" <gi...@apache.org> on 2026/09/08 10:02:59 UTC

[PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

youngrae81 opened a new pull request, #5466:
URL: https://github.com/apache/polaris/pull/5466

   `PageTokenUtil.decodePageRequest` decoded the client-supplied `pageToken` (Base64 + SMILE) without error handling. A token that is valid Base64 but not a serialized page token (garbage, truncated, or produced by an incompatible Polaris version) escaped as a Jackson 3 `JacksonException` or, for an unknown token type id, as `IllegalStateException`. Neither is covered by the REST exception mappers, so client input turned into a 500 with an ERROR-level stack trace.
   
   Wrap the decoding step and rethrow any failure as
   `IllegalArgumentException("Invalid page token")`, which the existing `IcebergExceptionMapper` maps to 400. The original exception is kept as the cause for debug logging.
   
   Adds a parameterized unit test covering the malformed-token shapes and an integration test asserting the HTTP 400 on `listNamespaces`.
   
   Fixes #5465
   
   <!--
   ๐Ÿ“ Describe what changes you're proposing, especially breaking or user-facing changes. 
   ๐Ÿ“– See https://github.com/apache/polaris/blob/main/CONTRIBUTING.md for more.
   -->
   
   ## Checklist
   - [ ] ๐Ÿ›ก๏ธ Don't disclose security issues! (contact security@apache.org)
   - [ ] ๐Ÿ”— Clearly explained why the changes are needed, or linked related issues: Fixes #
   - [ ] ๐Ÿงช Added/updated tests with good coverage, or manually tested (and explained how)
   - [ ] ๐Ÿ’ก Added comments for complex logic
   - [ ] ๐Ÿงพ Updated `CHANGELOG.md` (if needed)
   - [ ] ๐Ÿ“š Updated documentation in `site/content/in-dev/unreleased` (if needed)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3963718959


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -141,6 +140,25 @@ static PageToken decodePageRequest(
     }
   }
 
+  /**
+   * Decodes a client-supplied page token. Page tokens are opaque to clients, so any decoding
+   * failure is client input that cannot be interpreted and must surface as an {@link
+   * IllegalArgumentException} (mapped to HTTP 400), never as a server error.
+   */
+  private static PageToken deserializePageToken(String requestedPageToken) {
+    try {
+      var bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      return SMILE_MAPPER.readValue(bytes, PageToken.class);

Review Comment:
   Good catch, thank you.
   Reproduced: without pageSize the null token was returned to the caller, and with pageSize it failed with a NullPointerException outside the catch. The latest commit rejects a null result as an invalid page token, and the test class now has a "SMILE-encoded null" case plus a parameterized test covering pageSize absent, 0 and 5.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3960024640


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -111,6 +112,11 @@ public String toString() {
 
   private PageTokenUtil() {}
 
+  @VisibleForTesting

Review Comment:
   Agreed, removed in 04ed32a. The test now builds its own plain SmileMapper to craft non-PageToken payloads, so the production change is just the catch block.



##########
polaris-core/src/test/java/org/apache/polaris/core/persistence/pagination/MalformedPageTokenTest.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Map;
+import java.util.stream.Stream;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+/**
+ * A client-supplied {@code pageToken} that cannot be decoded must surface as an {@link
+ * IllegalArgumentException}, which the REST layer maps to HTTP 400. Any other exception type
+ * (Jackson decoding failures, {@link IllegalStateException} from the token-type registry) escapes
+ * the exception mappers and turns client input into an HTTP 500.
+ */
+class MalformedPageTokenTest {
+
+  @ParameterizedTest(name = "{0}")
+  @MethodSource
+  void malformedTokenIsRejectedAsBadRequest(String description, String serializedToken) {
+    assertThatThrownBy(() -> PageToken.build(serializedToken, null, () -> true))
+        .as(description)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  static Stream<Arguments> malformedTokenIsRejectedAsBadRequest() {
+    return Stream.of(
+        arguments("not base64", "%%%not-base64%%%"),
+        arguments("valid base64, not SMILE", "AAAA"),
+        arguments(
+            "valid base64, plain text",
+            encode("hello world".getBytes(java.nio.charset.StandardCharsets.UTF_8))),

Review Comment:
   Added in 04ed32a: a full single-bit-flip and 0x00/0xFF byte-overwrite sweep over a valid token, plus a seeded random corruption test (overwrite/truncate/insert/flip, 20k iterations) over tokens of different shapes. 
   
   A corrupted token may still decode, but must never fail with anything other than IllegalArgumentException.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r4007259965


##########
polaris-core/src/test/java/org/apache/polaris/core/persistence/pagination/MalformedPageTokenTest.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assertions.fail;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullSource;
+import org.junit.jupiter.params.provider.ValueSource;
+import tools.jackson.databind.ObjectMapper;
+import tools.jackson.dataformat.smile.SmileMapper;
+
+/**
+ * A client-supplied {@code pageToken} that cannot be decoded must surface as an {@link
+ * IllegalArgumentException}, which the REST layer maps to HTTP 400. Any other exception type
+ * (Jackson decoding failures, {@link IllegalStateException} from the token-type registry, parser
+ * errors on corrupted binary input) escapes the exception mappers and turns client input into an
+ * HTTP 500.
+ */
+class MalformedPageTokenTest {
+
+  /** Plain SMILE mapper to craft arbitrary payloads that are not {@link PageToken}s. */
+  private static final ObjectMapper RAW_SMILE = SmileMapper.builder().build();
+
+  @ParameterizedTest(name = "{0}")
+  @MethodSource
+  void malformedTokenIsRejectedAsBadRequest(String description, String serializedToken) {
+    assertThatThrownBy(() -> PageToken.build(serializedToken, null, () -> true))
+        .as(description)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  static Stream<Arguments> malformedTokenIsRejectedAsBadRequest() {
+    return Stream.of(
+        arguments("not base64", "%%%not-base64%%%"),
+        arguments("valid base64, not SMILE", "AAAA"),
+        arguments("valid base64, plain text", encode("hello world".getBytes(UTF_8))),
+        arguments(
+            "SMILE with unknown token type id",
+            smile(Map.of("p", 5, "v", Map.of("t", "no-such-type")))),
+        arguments("SMILE with wrong shape (array)", smile(new int[] {1, 2, 3})),
+        arguments("truncated valid token", encode(truncate(validTokenBytes()))),
+        // Found by fuzzing a valid token: the SMILE parser fails with
+        // ArrayIndexOutOfBoundsException rather than a Jackson exception.
+        arguments(
+            "corrupted SMILE that trips the parser itself", "OikKAfqAcNTKdvqAdEBlgGkoAWs8aKr7-w=="),
+        // Deserializes "successfully" to null, so the failure would otherwise surface later as a
+        // NullPointerException.
+        arguments("SMILE-encoded null", smile(null)));
+  }
+
+  /**
+   * The page size is applied to the decoded token after deserialization; a token that decodes to
+   * null must be rejected before that, with or without a page size.
+   */
+  @ParameterizedTest(name = "pageSize={0}")
+  @NullSource
+  @ValueSource(ints = {0, 5})
+  void nullTokenIsRejectedWithAndWithoutPageSize(Integer pageSize) {
+    String nullToken = smile(null);
+    assertThatThrownBy(() -> PageToken.build(nullToken, pageSize, () -> true))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  /**
+   * Byte-level corruption: flip every single bit of a valid token, one at a time, and overwrite
+   * every byte with 0x00 and 0xFF. A corrupted token is allowed to still decode (some flips land in
+   * payload bytes and yield a different but well-formed token), but it must never fail with
+   * anything other than {@link IllegalArgumentException}.
+   */
+  @Test
+  void corruptedTokenNeverEscapesAsServerError() {
+    byte[] valid = validTokenBytes();
+
+    Stream<byte[]> bitFlips =
+        IntStream.range(0, valid.length * 8)
+            .mapToObj(
+                bit -> {
+                  byte[] corrupted = valid.clone();
+                  corrupted[bit / 8] ^= (byte) (1 << (bit % 8));
+                  return corrupted;
+                });
+    Stream<byte[]> byteOverwrites =
+        IntStream.range(0, valid.length)
+            .boxed()
+            .flatMap(
+                i ->
+                    Stream.of((byte) 0x00, (byte) 0xFF)
+                        .map(
+                            b -> {
+                              byte[] corrupted = valid.clone();
+                              corrupted[i] = b;
+                              return corrupted;
+                            }));
+
+    Stream.concat(bitFlips, byteOverwrites)
+        .map(MalformedPageTokenTest::encode)
+        .forEach(MalformedPageTokenTest::assertDecodesOrIsBadRequest);
+  }
+
+  /**
+   * Random corruption with a fixed seed: overwrite, truncate, insert and flip bytes of valid tokens
+   * of different shapes. Deterministic, so a failure reproduces.
+   */
+  @Test
+  void randomlyCorruptedTokenNeverEscapesAsServerError() {
+    List<byte[]> seeds =
+        List.of(
+            validTokenBytes(),
+            tokenBytes(
+                ImmutablePageToken.builder()
+                    .pageSize(10)
+                    .value(EntityIdToken.fromEntityId(123456789L))
+                    .build()),
+            tokenBytes(
+                ImmutablePageToken.builder()
+                    .pageSize(1000)
+                    .value(
+                        ImmutableDummyTestToken.builder()
+                            .s("a".repeat(300))
+                            .i(Integer.MAX_VALUE)
+                            .build())
+                    .build()));
+    Random random = new Random(42);
+    for (int iteration = 0; iteration < 20_000; iteration++) {
+      byte[] valid = seeds.get(iteration % seeds.size());
+      byte[] corrupted;
+      switch (random.nextInt(4)) {
+        case 0 -> {
+          corrupted = valid.clone();
+          for (int k = 0, n = 1 + random.nextInt(3); k < n; k++) {
+            corrupted[random.nextInt(corrupted.length)] = (byte) random.nextInt(256);
+          }
+        }
+        case 1 -> corrupted = Arrays.copyOf(valid, random.nextInt(valid.length));
+        case 2 -> {
+          int at = random.nextInt(valid.length);
+          corrupted = new byte[valid.length + 1];
+          System.arraycopy(valid, 0, corrupted, 0, at);
+          corrupted[at] = (byte) random.nextInt(256);
+          System.arraycopy(valid, at, corrupted, at + 1, valid.length - at);
+        }
+        default -> {
+          corrupted = valid.clone();
+          int at = random.nextInt(corrupted.length);
+          corrupted[at] ^= (byte) (1 << random.nextInt(8));
+          corrupted[(at + 1) % corrupted.length] = (byte) 0xFF;
+        }
+      }
+      assertDecodesOrIsBadRequest(encode(corrupted));
+    }
+  }
+
+  private static void assertDecodesOrIsBadRequest(String token) {
+    try {
+      PageToken decoded = PageToken.build(token, null, () -> true);
+      assertThat(decoded).as("token %s", token).isNotNull();

Review Comment:
   Good point, thanks. 
   After the null-token rejection PageToken.build can no longer return null, so that assertion could never fail. 
   The latest commit removes it: a corrupted token that still decodes needs no assertion, and the helper now only checks the failure path (IllegalArgumentException with "Invalid page token"). For reference, in the exploratory fuzz run about a quarter of random corruptions still decoded to a well-formed token.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r4010738828


##########
integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java:
##########
@@ -2862,6 +2862,27 @@ public void testPaginatedListNamespaces() {
     }
   }
 
+  @Test
+  public void testMalformedPageTokenIsBadRequest() {
+    // Valid url-safe base64 that is not a serialized page token. Client input, so the server
+    // must answer 400, not 500.
+    for (String badToken : List.of("AAAA", "aGVsbG8gd29ybGQ=", "%%%not-base64%%%")) {

Review Comment:
   Done in the latest commit, converted to @ParameterizedTest with the three token shapes as @ValueSource



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3960008687


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -141,6 +146,23 @@ static PageToken decodePageRequest(
     }
   }
 
+  /**
+   * Decodes a client-supplied page token. Page tokens are opaque to clients, so any decoding
+   * failure is client input that cannot be interpreted and must surface as an {@link
+   * IllegalArgumentException} (mapped to HTTP 400), never as a server error.
+   */
+  private static PageToken deserializePageToken(String requestedPageToken) {
+    try {
+      var bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      return SMILE_MAPPER.readValue(bytes, PageToken.class);
+    } catch (IllegalArgumentException | IllegalStateException | JacksonException e) {

Review Comment:
   Good catch, thanks. Fuzzing a valid token reproduced it: 4 of 60k random corruptions escaped as ArrayIndexOutOfBoundsException. Widened to RuntimeException in 04ed32a and added the fuzz-found  token as a fixed regression case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "flyingImer (via GitHub)" <gi...@apache.org>.
flyingImer commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3961445473


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -141,6 +140,25 @@ static PageToken decodePageRequest(
     }
   }
 
+  /**
+   * Decodes a client-supplied page token. Page tokens are opaque to clients, so any decoding
+   * failure is client input that cannot be interpreted and must surface as an {@link
+   * IllegalArgumentException} (mapped to HTTP 400), never as a server error.
+   */
+  private static PageToken deserializePageToken(String requestedPageToken) {
+    try {
+      var bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      return SMILE_MAPPER.readValue(bytes, PageToken.class);

Review Comment:
   Could we also reject a null result here? A SMILE root null deserializes successfully, so this catch never runs IIUC. With pageSize supplied, decodePageRequest then dereferences that null outside the catch and the request still returns 500. Please reject it as an invalid page token and add a regression case confirming 400 both with and without pageSize.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3957927913


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -141,6 +146,23 @@ static PageToken decodePageRequest(
     }
   }
 
+  /**
+   * Decodes a client-supplied page token. Page tokens are opaque to clients, so any decoding
+   * failure is client input that cannot be interpreted and must surface as an {@link
+   * IllegalArgumentException} (mapped to HTTP 400), never as a server error.
+   */
+  private static PageToken deserializePageToken(String requestedPageToken) {

Review Comment:
   Maybe we could reuse the Polaris standard serialization with Jackson 3?



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -141,6 +146,23 @@ static PageToken decodePageRequest(
     }
   }
 
+  /**
+   * Decodes a client-supplied page token. Page tokens are opaque to clients, so any decoding
+   * failure is client input that cannot be interpreted and must surface as an {@link
+   * IllegalArgumentException} (mapped to HTTP 400), never as a server error.
+   */
+  private static PageToken deserializePageToken(String requestedPageToken) {
+    try {
+      var bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      return SMILE_MAPPER.readValue(bytes, PageToken.class);
+    } catch (IllegalArgumentException | IllegalStateException | JacksonException e) {

Review Comment:
   I believe this catch doesn't get every exception a corrupted SMILE payload can throw.
   
   For instance, fuzzing a valid token throws `ArrayIndexOutOfBoundsException`, which isn't caught here and falls through to `IcebergExceptionMapper`'s default 500.
   
   I suggest widening to `catch (RuntimeException e)` instead of enumerating types.



##########
polaris-core/src/test/java/org/apache/polaris/core/persistence/pagination/MalformedPageTokenTest.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Map;
+import java.util.stream.Stream;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+/**
+ * A client-supplied {@code pageToken} that cannot be decoded must surface as an {@link
+ * IllegalArgumentException}, which the REST layer maps to HTTP 400. Any other exception type
+ * (Jackson decoding failures, {@link IllegalStateException} from the token-type registry) escapes
+ * the exception mappers and turns client input into an HTTP 500.
+ */
+class MalformedPageTokenTest {
+
+  @ParameterizedTest(name = "{0}")
+  @MethodSource
+  void malformedTokenIsRejectedAsBadRequest(String description, String serializedToken) {
+    assertThatThrownBy(() -> PageToken.build(serializedToken, null, () -> true))
+        .as(description)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  static Stream<Arguments> malformedTokenIsRejectedAsBadRequest() {
+    return Stream.of(
+        arguments("not base64", "%%%not-base64%%%"),
+        arguments("valid base64, not SMILE", "AAAA"),
+        arguments(
+            "valid base64, plain text",
+            encode("hello world".getBytes(java.nio.charset.StandardCharsets.UTF_8))),

Review Comment:
   Maybe we should cover the byte-level/bit-flip corruption?



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -111,6 +112,11 @@ public String toString() {
 
   private PageTokenUtil() {}
 
+  @VisibleForTesting

Review Comment:
   Doesn't it overlap with the existing `serializePageToken()`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3960024640


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -111,6 +112,11 @@ public String toString() {
 
   private PageTokenUtil() {}
 
+  @VisibleForTesting

Review Comment:
   Agreed, removed in 7beee54. The test now builds its own plain SmileMapper to craft non-PageToken payloads, so the production change is just the catch block.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3960032985


##########
polaris-core/src/test/java/org/apache/polaris/core/persistence/pagination/MalformedPageTokenTest.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Map;
+import java.util.stream.Stream;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+/**
+ * A client-supplied {@code pageToken} that cannot be decoded must surface as an {@link
+ * IllegalArgumentException}, which the REST layer maps to HTTP 400. Any other exception type
+ * (Jackson decoding failures, {@link IllegalStateException} from the token-type registry) escapes
+ * the exception mappers and turns client input into an HTTP 500.
+ */
+class MalformedPageTokenTest {
+
+  @ParameterizedTest(name = "{0}")
+  @MethodSource
+  void malformedTokenIsRejectedAsBadRequest(String description, String serializedToken) {
+    assertThatThrownBy(() -> PageToken.build(serializedToken, null, () -> true))
+        .as(description)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  static Stream<Arguments> malformedTokenIsRejectedAsBadRequest() {
+    return Stream.of(
+        arguments("not base64", "%%%not-base64%%%"),
+        arguments("valid base64, not SMILE", "AAAA"),
+        arguments(
+            "valid base64, plain text",
+            encode("hello world".getBytes(java.nio.charset.StandardCharsets.UTF_8))),

Review Comment:
   Added in 4a9ffbd: a full single-bit-flip and 0x00/0xFF byte-overwrite sweep over a valid token, plus a seeded random corruption test (overwrite/truncate/insert/flip, 20k iterations) over tokens of different shapes. 
   
   A corrupted token may still decode, but must never fail with anything other than IllegalArgumentException.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3960017579


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -141,6 +146,23 @@ static PageToken decodePageRequest(
     }
   }
 
+  /**
+   * Decodes a client-supplied page token. Page tokens are opaque to clients, so any decoding
+   * failure is client input that cannot be interpreted and must surface as an {@link
+   * IllegalArgumentException} (mapped to HTTP 400), never as a server error.
+   */
+  private static PageToken deserializePageToken(String requestedPageToken) {

Review Comment:
   I looked at PolarisObjectMapperUtil, but it is a Jackson 2 JSON mapper, while page tokens are binary SMILE via Jackson 3 (the SmileMapper here predates this PR). 
   I kept it as is; happy to switch if there is a shared Jackson 3 mapper you had in mind.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3960024640


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -111,6 +112,11 @@ public String toString() {
 
   private PageTokenUtil() {}
 
+  @VisibleForTesting

Review Comment:
   Agreed, removed in 4a9ffbd. The test now builds its own plain SmileMapper to craft non-PageToken payloads, so the production change is just the catch block.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3960032985


##########
polaris-core/src/test/java/org/apache/polaris/core/persistence/pagination/MalformedPageTokenTest.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Map;
+import java.util.stream.Stream;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+/**
+ * A client-supplied {@code pageToken} that cannot be decoded must surface as an {@link
+ * IllegalArgumentException}, which the REST layer maps to HTTP 400. Any other exception type
+ * (Jackson decoding failures, {@link IllegalStateException} from the token-type registry) escapes
+ * the exception mappers and turns client input into an HTTP 500.
+ */
+class MalformedPageTokenTest {
+
+  @ParameterizedTest(name = "{0}")
+  @MethodSource
+  void malformedTokenIsRejectedAsBadRequest(String description, String serializedToken) {
+    assertThatThrownBy(() -> PageToken.build(serializedToken, null, () -> true))
+        .as(description)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  static Stream<Arguments> malformedTokenIsRejectedAsBadRequest() {
+    return Stream.of(
+        arguments("not base64", "%%%not-base64%%%"),
+        arguments("valid base64, not SMILE", "AAAA"),
+        arguments(
+            "valid base64, plain text",
+            encode("hello world".getBytes(java.nio.charset.StandardCharsets.UTF_8))),

Review Comment:
   Added in 7beee54: a full single-bit-flip and 0x00/0xFF byte-overwrite sweep over a valid token, plus a seeded random corruption test (overwrite/truncate/insert/flip, 20k iterations) over tokens of different shapes. 
   
   A corrupted token may still decode, but must never fail with anything other than IllegalArgumentException.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b merged PR #5466:
URL: https://github.com/apache/polaris/pull/5466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on PR #5466:
URL: https://github.com/apache/polaris/pull/5466#issuecomment-5657556012

   @jbonofre @flyingImer Both review rounds are addressed (RuntimeException catch, fuzz-found regression case, null-token rejection) and the branch is rebased on main. 
   CI has not run yet because it is waiting for a maintainer to approve the workflow. Could one of you approve the run and take another look? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r4007292768


##########
integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java:
##########
@@ -2862,6 +2862,27 @@ public void testPaginatedListNamespaces() {
     }
   }
 
+  @Test
+  public void testMalformedPageTokenIsBadRequest() {
+    // Valid url-safe base64 that is not a serialized page token. Client input, so the server
+    // must answer 400, not 500.
+    for (String badToken : List.of("AAAA", "aGVsbG8gd29ybGQ=", "%%%not-base64%%%")) {

Review Comment:
   nit: convert to `@ParameterizedTest`?



##########
polaris-core/src/test/java/org/apache/polaris/core/persistence/pagination/MalformedPageTokenTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assertions.catchThrowable;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullSource;
+import org.junit.jupiter.params.provider.ValueSource;
+import tools.jackson.databind.ObjectMapper;
+import tools.jackson.dataformat.smile.SmileMapper;
+
+/**
+ * A client-supplied {@code pageToken} that cannot be decoded must surface as an {@link
+ * IllegalArgumentException}, which the REST layer maps to HTTP 400. Any other exception type
+ * (Jackson decoding failures, {@link IllegalStateException} from the token-type registry, parser
+ * errors on corrupted binary input) escapes the exception mappers and turns client input into an
+ * HTTP 500.
+ */
+class MalformedPageTokenTest {
+
+  /** Plain SMILE mapper to craft arbitrary payloads that are not {@link PageToken}s. */
+  private static final ObjectMapper RAW_SMILE = SmileMapper.builder().build();
+
+  @ParameterizedTest(name = "{0}")
+  @MethodSource
+  void malformedTokenIsRejectedAsBadRequest(String description, String serializedToken) {
+    assertThatThrownBy(() -> PageToken.build(serializedToken, null, () -> true))
+        .as(description)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  static Stream<Arguments> malformedTokenIsRejectedAsBadRequest() {
+    return Stream.of(
+        arguments("not base64", "%%%not-base64%%%"),
+        arguments("valid base64, not SMILE", "AAAA"),
+        arguments("valid base64, plain text", encode("hello world".getBytes(UTF_8))),
+        arguments(
+            "SMILE with unknown token type id",
+            smile(Map.of("p", 5, "v", Map.of("t", "no-such-type")))),
+        arguments("SMILE with wrong shape (array)", smile(new int[] {1, 2, 3})),
+        arguments("truncated valid token", encode(truncate(validTokenBytes()))),
+        // Found by fuzzing a valid token: the SMILE parser fails with
+        // ArrayIndexOutOfBoundsException rather than a Jackson exception.
+        arguments(
+            "corrupted SMILE that trips the parser itself", "OikKAfqAcNTKdvqAdEBlgGkoAWs8aKr7-w=="),
+        // Deserializes "successfully" to null, so the failure would otherwise surface later as a
+        // NullPointerException.
+        arguments("SMILE-encoded null", smile(null)));
+  }
+
+  /**
+   * The page size is applied to the decoded token after deserialization; a token that decodes to
+   * null must be rejected before that, with or without a page size.
+   */
+  @ParameterizedTest(name = "pageSize={0}")
+  @NullSource
+  @ValueSource(ints = {0, 5})
+  void nullTokenIsRejectedWithAndWithoutPageSize(Integer pageSize) {
+    String nullToken = smile(null);
+    assertThatThrownBy(() -> PageToken.build(nullToken, pageSize, () -> true))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  /**
+   * Byte-level corruption: flip every single bit of a valid token, one at a time, and overwrite
+   * every byte with 0x00 and 0xFF. A corrupted token is allowed to still decode (some flips land in
+   * payload bytes and yield a different but well-formed token), but it must never fail with
+   * anything other than {@link IllegalArgumentException}.
+   */
+  @Test
+  void corruptedTokenNeverEscapesAsServerError() {
+    byte[] valid = validTokenBytes();
+
+    Stream<byte[]> bitFlips =
+        IntStream.range(0, valid.length * 8)
+            .mapToObj(
+                bit -> {
+                  byte[] corrupted = valid.clone();
+                  corrupted[bit / 8] ^= (byte) (1 << (bit % 8));
+                  return corrupted;
+                });
+    Stream<byte[]> byteOverwrites =
+        IntStream.range(0, valid.length)
+            .boxed()
+            .flatMap(
+                i ->
+                    Stream.of((byte) 0x00, (byte) 0xFF)
+                        .map(
+                            b -> {
+                              byte[] corrupted = valid.clone();
+                              corrupted[i] = b;
+                              return corrupted;
+                            }));
+
+    Stream.concat(bitFlips, byteOverwrites)
+        .map(MalformedPageTokenTest::encode)
+        .forEach(MalformedPageTokenTest::assertDecodesOrIsBadRequest);
+  }
+
+  /**
+   * Random corruption with a fixed seed: overwrite, truncate, insert and flip bytes of valid tokens
+   * of different shapes. Deterministic, so a failure reproduces.
+   */
+  @Test
+  void randomlyCorruptedTokenNeverEscapesAsServerError() {
+    List<byte[]> seeds =
+        List.of(
+            validTokenBytes(),
+            tokenBytes(
+                ImmutablePageToken.builder()
+                    .pageSize(10)
+                    .value(EntityIdToken.fromEntityId(123456789L))
+                    .build()),
+            tokenBytes(
+                ImmutablePageToken.builder()
+                    .pageSize(1000)
+                    .value(
+                        ImmutableDummyTestToken.builder()
+                            .s("a".repeat(300))
+                            .i(Integer.MAX_VALUE)
+                            .build())
+                    .build()));
+    Random random = new Random(42);
+    for (int iteration = 0; iteration < 20_000; iteration++) {
+      byte[] valid = seeds.get(iteration % seeds.size());
+      byte[] corrupted;
+      switch (random.nextInt(4)) {
+        case 0 -> {
+          corrupted = valid.clone();
+          for (int k = 0, n = 1 + random.nextInt(3); k < n; k++) {
+            corrupted[random.nextInt(corrupted.length)] = (byte) random.nextInt(256);
+          }
+        }
+        case 1 -> corrupted = Arrays.copyOf(valid, random.nextInt(valid.length));
+        case 2 -> {
+          int at = random.nextInt(valid.length);
+          corrupted = new byte[valid.length + 1];
+          System.arraycopy(valid, 0, corrupted, 0, at);
+          corrupted[at] = (byte) random.nextInt(256);
+          System.arraycopy(valid, at, corrupted, at + 1, valid.length - at);
+        }
+        default -> {
+          corrupted = valid.clone();
+          int at = random.nextInt(corrupted.length);
+          corrupted[at] ^= (byte) (1 << random.nextInt(8));
+          corrupted[(at + 1) % corrupted.length] = (byte) 0xFF;
+        }
+      }
+      assertDecodesOrIsBadRequest(encode(corrupted));

Review Comment:
   nit: TBH, I'm not sure `randomlyCorruptedTokenNeverEscapesAsServerError` and `corruptedTokenNeverEscapesAsServerError` add that much value ๐Ÿค” 
   
   The tests are good, but what do they cover? Normal parser exceptions are covered by other tests. These tests can potentially uncover some odd error cases in the _parser code_, which are not handled by existing exception call paths in _Polaris code_... However, it does not happen in practice ATM, as far as I can tell... and it is not likely to happen in the future. If the parser through an unchecked exception other than a `RuntimeException` it must be an `Error`... I do not think it is a likely scenario, though.
   
   In any case a 500 response on `Error` is probably going to be appropriate.
   
   IMHO, it might be best to remove them to reduce test complexity. However, I'm ok to keep they if you prefer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3960008687


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -141,6 +146,23 @@ static PageToken decodePageRequest(
     }
   }
 
+  /**
+   * Decodes a client-supplied page token. Page tokens are opaque to clients, so any decoding
+   * failure is client input that cannot be interpreted and must surface as an {@link
+   * IllegalArgumentException} (mapped to HTTP 400), never as a server error.
+   */
+  private static PageToken deserializePageToken(String requestedPageToken) {
+    try {
+      var bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      return SMILE_MAPPER.readValue(bytes, PageToken.class);
+    } catch (IllegalArgumentException | IllegalStateException | JacksonException e) {

Review Comment:
   Good catch, thanks. Fuzzing a valid token reproduced it: 4 of 60k random corruptions escaped as ArrayIndexOutOfBoundsException. Widened to RuntimeException in 7beee54 and added the fuzz-found  token as a fixed regression case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r4007038935


##########
polaris-core/src/test/java/org/apache/polaris/core/persistence/pagination/MalformedPageTokenTest.java:
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assertions.fail;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullSource;
+import org.junit.jupiter.params.provider.ValueSource;
+import tools.jackson.databind.ObjectMapper;
+import tools.jackson.dataformat.smile.SmileMapper;
+
+/**
+ * A client-supplied {@code pageToken} that cannot be decoded must surface as an {@link
+ * IllegalArgumentException}, which the REST layer maps to HTTP 400. Any other exception type
+ * (Jackson decoding failures, {@link IllegalStateException} from the token-type registry, parser
+ * errors on corrupted binary input) escapes the exception mappers and turns client input into an
+ * HTTP 500.
+ */
+class MalformedPageTokenTest {
+
+  /** Plain SMILE mapper to craft arbitrary payloads that are not {@link PageToken}s. */
+  private static final ObjectMapper RAW_SMILE = SmileMapper.builder().build();
+
+  @ParameterizedTest(name = "{0}")
+  @MethodSource
+  void malformedTokenIsRejectedAsBadRequest(String description, String serializedToken) {
+    assertThatThrownBy(() -> PageToken.build(serializedToken, null, () -> true))
+        .as(description)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  static Stream<Arguments> malformedTokenIsRejectedAsBadRequest() {
+    return Stream.of(
+        arguments("not base64", "%%%not-base64%%%"),
+        arguments("valid base64, not SMILE", "AAAA"),
+        arguments("valid base64, plain text", encode("hello world".getBytes(UTF_8))),
+        arguments(
+            "SMILE with unknown token type id",
+            smile(Map.of("p", 5, "v", Map.of("t", "no-such-type")))),
+        arguments("SMILE with wrong shape (array)", smile(new int[] {1, 2, 3})),
+        arguments("truncated valid token", encode(truncate(validTokenBytes()))),
+        // Found by fuzzing a valid token: the SMILE parser fails with
+        // ArrayIndexOutOfBoundsException rather than a Jackson exception.
+        arguments(
+            "corrupted SMILE that trips the parser itself", "OikKAfqAcNTKdvqAdEBlgGkoAWs8aKr7-w=="),
+        // Deserializes "successfully" to null, so the failure would otherwise surface later as a
+        // NullPointerException.
+        arguments("SMILE-encoded null", smile(null)));
+  }
+
+  /**
+   * The page size is applied to the decoded token after deserialization; a token that decodes to
+   * null must be rejected before that, with or without a page size.
+   */
+  @ParameterizedTest(name = "pageSize={0}")
+  @NullSource
+  @ValueSource(ints = {0, 5})
+  void nullTokenIsRejectedWithAndWithoutPageSize(Integer pageSize) {
+    String nullToken = smile(null);
+    assertThatThrownBy(() -> PageToken.build(nullToken, pageSize, () -> true))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  /**
+   * Byte-level corruption: flip every single bit of a valid token, one at a time, and overwrite
+   * every byte with 0x00 and 0xFF. A corrupted token is allowed to still decode (some flips land in
+   * payload bytes and yield a different but well-formed token), but it must never fail with
+   * anything other than {@link IllegalArgumentException}.
+   */
+  @Test
+  void corruptedTokenNeverEscapesAsServerError() {
+    byte[] valid = validTokenBytes();
+
+    Stream<byte[]> bitFlips =
+        IntStream.range(0, valid.length * 8)
+            .mapToObj(
+                bit -> {
+                  byte[] corrupted = valid.clone();
+                  corrupted[bit / 8] ^= (byte) (1 << (bit % 8));
+                  return corrupted;
+                });
+    Stream<byte[]> byteOverwrites =
+        IntStream.range(0, valid.length)
+            .boxed()
+            .flatMap(
+                i ->
+                    Stream.of((byte) 0x00, (byte) 0xFF)
+                        .map(
+                            b -> {
+                              byte[] corrupted = valid.clone();
+                              corrupted[i] = b;
+                              return corrupted;
+                            }));
+
+    Stream.concat(bitFlips, byteOverwrites)
+        .map(MalformedPageTokenTest::encode)
+        .forEach(MalformedPageTokenTest::assertDecodesOrIsBadRequest);
+  }
+
+  /**
+   * Random corruption with a fixed seed: overwrite, truncate, insert and flip bytes of valid tokens
+   * of different shapes. Deterministic, so a failure reproduces.
+   */
+  @Test
+  void randomlyCorruptedTokenNeverEscapesAsServerError() {
+    List<byte[]> seeds =
+        List.of(
+            validTokenBytes(),
+            tokenBytes(
+                ImmutablePageToken.builder()
+                    .pageSize(10)
+                    .value(EntityIdToken.fromEntityId(123456789L))
+                    .build()),
+            tokenBytes(
+                ImmutablePageToken.builder()
+                    .pageSize(1000)
+                    .value(
+                        ImmutableDummyTestToken.builder()
+                            .s("a".repeat(300))
+                            .i(Integer.MAX_VALUE)
+                            .build())
+                    .build()));
+    Random random = new Random(42);
+    for (int iteration = 0; iteration < 20_000; iteration++) {
+      byte[] valid = seeds.get(iteration % seeds.size());
+      byte[] corrupted;
+      switch (random.nextInt(4)) {
+        case 0 -> {
+          corrupted = valid.clone();
+          for (int k = 0, n = 1 + random.nextInt(3); k < n; k++) {
+            corrupted[random.nextInt(corrupted.length)] = (byte) random.nextInt(256);
+          }
+        }
+        case 1 -> corrupted = Arrays.copyOf(valid, random.nextInt(valid.length));
+        case 2 -> {
+          int at = random.nextInt(valid.length);
+          corrupted = new byte[valid.length + 1];
+          System.arraycopy(valid, 0, corrupted, 0, at);
+          corrupted[at] = (byte) random.nextInt(256);
+          System.arraycopy(valid, at, corrupted, at + 1, valid.length - at);
+        }
+        default -> {
+          corrupted = valid.clone();
+          int at = random.nextInt(corrupted.length);
+          corrupted[at] ^= (byte) (1 << random.nextInt(8));
+          corrupted[(at + 1) % corrupted.length] = (byte) 0xFF;
+        }
+      }
+      assertDecodesOrIsBadRequest(encode(corrupted));
+    }
+  }
+
+  private static void assertDecodesOrIsBadRequest(String token) {
+    try {
+      PageToken decoded = PageToken.build(token, null, () -> true);
+      assertThat(decoded).as("token %s", token).isNotNull();

Review Comment:
   When do we expect this assertion to pass?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r4010743136


##########
polaris-core/src/test/java/org/apache/polaris/core/persistence/pagination/MalformedPageTokenTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence.pagination;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assertions.catchThrowable;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullSource;
+import org.junit.jupiter.params.provider.ValueSource;
+import tools.jackson.databind.ObjectMapper;
+import tools.jackson.dataformat.smile.SmileMapper;
+
+/**
+ * A client-supplied {@code pageToken} that cannot be decoded must surface as an {@link
+ * IllegalArgumentException}, which the REST layer maps to HTTP 400. Any other exception type
+ * (Jackson decoding failures, {@link IllegalStateException} from the token-type registry, parser
+ * errors on corrupted binary input) escapes the exception mappers and turns client input into an
+ * HTTP 500.
+ */
+class MalformedPageTokenTest {
+
+  /** Plain SMILE mapper to craft arbitrary payloads that are not {@link PageToken}s. */
+  private static final ObjectMapper RAW_SMILE = SmileMapper.builder().build();
+
+  @ParameterizedTest(name = "{0}")
+  @MethodSource
+  void malformedTokenIsRejectedAsBadRequest(String description, String serializedToken) {
+    assertThatThrownBy(() -> PageToken.build(serializedToken, null, () -> true))
+        .as(description)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  static Stream<Arguments> malformedTokenIsRejectedAsBadRequest() {
+    return Stream.of(
+        arguments("not base64", "%%%not-base64%%%"),
+        arguments("valid base64, not SMILE", "AAAA"),
+        arguments("valid base64, plain text", encode("hello world".getBytes(UTF_8))),
+        arguments(
+            "SMILE with unknown token type id",
+            smile(Map.of("p", 5, "v", Map.of("t", "no-such-type")))),
+        arguments("SMILE with wrong shape (array)", smile(new int[] {1, 2, 3})),
+        arguments("truncated valid token", encode(truncate(validTokenBytes()))),
+        // Found by fuzzing a valid token: the SMILE parser fails with
+        // ArrayIndexOutOfBoundsException rather than a Jackson exception.
+        arguments(
+            "corrupted SMILE that trips the parser itself", "OikKAfqAcNTKdvqAdEBlgGkoAWs8aKr7-w=="),
+        // Deserializes "successfully" to null, so the failure would otherwise surface later as a
+        // NullPointerException.
+        arguments("SMILE-encoded null", smile(null)));
+  }
+
+  /**
+   * The page size is applied to the decoded token after deserialization; a token that decodes to
+   * null must be rejected before that, with or without a page size.
+   */
+  @ParameterizedTest(name = "pageSize={0}")
+  @NullSource
+  @ValueSource(ints = {0, 5})
+  void nullTokenIsRejectedWithAndWithoutPageSize(Integer pageSize) {
+    String nullToken = smile(null);
+    assertThatThrownBy(() -> PageToken.build(nullToken, pageSize, () -> true))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("Invalid page token");
+  }
+
+  /**
+   * Byte-level corruption: flip every single bit of a valid token, one at a time, and overwrite
+   * every byte with 0x00 and 0xFF. A corrupted token is allowed to still decode (some flips land in
+   * payload bytes and yield a different but well-formed token), but it must never fail with
+   * anything other than {@link IllegalArgumentException}.
+   */
+  @Test
+  void corruptedTokenNeverEscapesAsServerError() {
+    byte[] valid = validTokenBytes();
+
+    Stream<byte[]> bitFlips =
+        IntStream.range(0, valid.length * 8)
+            .mapToObj(
+                bit -> {
+                  byte[] corrupted = valid.clone();
+                  corrupted[bit / 8] ^= (byte) (1 << (bit % 8));
+                  return corrupted;
+                });
+    Stream<byte[]> byteOverwrites =
+        IntStream.range(0, valid.length)
+            .boxed()
+            .flatMap(
+                i ->
+                    Stream.of((byte) 0x00, (byte) 0xFF)
+                        .map(
+                            b -> {
+                              byte[] corrupted = valid.clone();
+                              corrupted[i] = b;
+                              return corrupted;
+                            }));
+
+    Stream.concat(bitFlips, byteOverwrites)
+        .map(MalformedPageTokenTest::encode)
+        .forEach(MalformedPageTokenTest::assertDecodesOrIsBadRequest);
+  }
+
+  /**
+   * Random corruption with a fixed seed: overwrite, truncate, insert and flip bytes of valid tokens
+   * of different shapes. Deterministic, so a failure reproduces.
+   */
+  @Test
+  void randomlyCorruptedTokenNeverEscapesAsServerError() {
+    List<byte[]> seeds =
+        List.of(
+            validTokenBytes(),
+            tokenBytes(
+                ImmutablePageToken.builder()
+                    .pageSize(10)
+                    .value(EntityIdToken.fromEntityId(123456789L))
+                    .build()),
+            tokenBytes(
+                ImmutablePageToken.builder()
+                    .pageSize(1000)
+                    .value(
+                        ImmutableDummyTestToken.builder()
+                            .s("a".repeat(300))
+                            .i(Integer.MAX_VALUE)
+                            .build())
+                    .build()));
+    Random random = new Random(42);
+    for (int iteration = 0; iteration < 20_000; iteration++) {
+      byte[] valid = seeds.get(iteration % seeds.size());
+      byte[] corrupted;
+      switch (random.nextInt(4)) {
+        case 0 -> {
+          corrupted = valid.clone();
+          for (int k = 0, n = 1 + random.nextInt(3); k < n; k++) {
+            corrupted[random.nextInt(corrupted.length)] = (byte) random.nextInt(256);
+          }
+        }
+        case 1 -> corrupted = Arrays.copyOf(valid, random.nextInt(valid.length));
+        case 2 -> {
+          int at = random.nextInt(valid.length);
+          corrupted = new byte[valid.length + 1];
+          System.arraycopy(valid, 0, corrupted, 0, at);
+          corrupted[at] = (byte) random.nextInt(256);
+          System.arraycopy(valid, at, corrupted, at + 1, valid.length - at);
+        }
+        default -> {
+          corrupted = valid.clone();
+          int at = random.nextInt(corrupted.length);
+          corrupted[at] ^= (byte) (1 << random.nextInt(8));
+          corrupted[(at + 1) % corrupted.length] = (byte) 0xFF;
+        }
+      }
+      assertDecodesOrIsBadRequest(encode(corrupted));

Review Comment:
   Agreed, removed both sweeps in the latest commit. 
   With the decoder catching RuntimeException, any parser failure already maps to 400, so they could not surface a new failure class. I kept the fuzz-found ArrayIndexOutOfBoundsException token as a fixed regression case, which is what jbonofre's original comment was about.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: return 400 instead of 500 for malformed pageToken on list endpoints [polaris]

Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5466:
URL: https://github.com/apache/polaris/pull/5466#discussion_r3960008687


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java:
##########
@@ -141,6 +146,23 @@ static PageToken decodePageRequest(
     }
   }
 
+  /**
+   * Decodes a client-supplied page token. Page tokens are opaque to clients, so any decoding
+   * failure is client input that cannot be interpreted and must surface as an {@link
+   * IllegalArgumentException} (mapped to HTTP 400), never as a server error.
+   */
+  private static PageToken deserializePageToken(String requestedPageToken) {
+    try {
+      var bytes = Base64.getUrlDecoder().decode(requestedPageToken);
+      return SMILE_MAPPER.readValue(bytes, PageToken.class);
+    } catch (IllegalArgumentException | IllegalStateException | JacksonException e) {

Review Comment:
   Good catch, thanks. Fuzzing a valid token reproduced it: 4 of 60k random corruptions escaped as ArrayIndexOutOfBoundsException. Widened to RuntimeException in 4a9ffbd and added the fuzz-found  token as a fixed regression case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org