You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "iprithv (via GitHub)" <gi...@apache.org> on 2026/09/28 18:03:56 UTC

[PR] Count decoded length in IndexKey.skip [polaris]

iprithv opened a new pull request, #5636:
URL: https://github.com/apache/polaris/pull/5636

   
   `IndexKey.skip` now measures decoded key length the same way as `deserializeKey`. Escaped `0x01` / `0x02` bytes count as one decoded unit each.
   
   ## Why
   
   `skip` previously counted two units per escaped byte against the same 500 limit. A key whose decoded length was under the limit (for example 251 bytes of `0x01`) could be constructed, serialized, and deserialized, then throw in `skip`. `IndexImpl` calls `skip` for every element while deserializing, so one such entry left the index container unreadable on later loads. NoSQL only.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [NoSQL] Count decoded length in IndexKey.skip [polaris]

Posted by "iprithv (via GitHub)" <gi...@apache.org>.
iprithv commented on PR #5636:
URL: https://github.com/apache/polaris/pull/5636#issuecomment-5920578440

   Thanks @dimas-b @snazy @jbonofre for the review. 
   
   Updated the tests based on your comments:
   
   * `keyLengthGood` / `keyTooLong` now cover `skip`, including escaped keys at and beyond `MAX_LENGTH`
   * `keySerializationRoundTrip` now checks the buffer advance from `skip`
   * `skipRejectsDecodedLengthOverMax` covers decoded length 501
   
   PTAL.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [NoSQL] Count decoded length in IndexKey.skip [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #5636:
URL: https://github.com/apache/polaris/pull/5636#discussion_r4125779175


##########
persistence/nosql/persistence/api/src/test/java/org/apache/polaris/persistence/nosql/api/index/TestIndexKey.java:
##########
@@ -80,6 +82,41 @@ static Stream<String> keyTooLong() {
         STRING_100 + STRING_100 + STRING_100 + STRING_100 + STRING_100 + STRING_100);
   }
 
+  /**
+   * Keys whose decoded length is under {@link IndexKey#MAX_LENGTH} but that expand when escaped
+   * must still be skippable. {@code skip} used to count two units per escaped byte and reject these
+   * keys after they had already been written.
+   */
+  @ParameterizedTest
+  @ValueSource(chars = {'\u0001', '\u0002'})
+  void skipAcceptsKeysThatExpandWhenEscaped(char escapedChar) {
+    // Decoded length 251: under MAX_LENGTH, but 251 escape pairs used to push skip's counter past
+    // 500 (2 * 251) and throw while deserializeKey still succeeded.
+    var indexKey = key(String.valueOf(escapedChar).repeat(251));
+    var buffer = ByteBuffer.allocate(indexKey.serializedSize());
+    indexKey.serialize(buffer).flip();
+
+    assertThat(deserializeKey(buffer.duplicate())).isEqualTo(indexKey);
+    var skipped = buffer.duplicate();
+    skip(skipped);
+    assertThat(skipped.remaining()).isZero();
+  }
+
+  @Test
+  void skipAcceptsMixedEscapedKeyNearLimit() {
+    // 250 escaped bytes + one ordinary byte: decoded length 251. The old skip counter would reach
+    // 501 (2 * 250 + 1) and throw.
+    var raw = "\u0001".repeat(250) + "a";

Review Comment:
   This yields only 251 chars, which is not "near" the limit (according to the method name) 🤔 Should we go all the way to `MAX_LENGTH`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [NoSQL] Count decoded length in IndexKey.skip [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #5636:
URL: https://github.com/apache/polaris/pull/5636#discussion_r4150376247


##########
persistence/nosql/persistence/api/src/test/java/org/apache/polaris/persistence/nosql/api/index/TestIndexKey.java:
##########
@@ -77,7 +95,23 @@ void keyTooLong(String value) {
   static Stream<String> keyTooLong() {
     return Stream.of(
         STRING_100 + STRING_100 + STRING_100 + STRING_100 + STRING_100 + "x",
-        STRING_100 + STRING_100 + STRING_100 + STRING_100 + STRING_100 + STRING_100);
+        STRING_100 + STRING_100 + STRING_100 + STRING_100 + STRING_100 + STRING_100,
+        "\u0001".repeat(IndexKey.MAX_LENGTH + 1),
+        "\u0002".repeat(IndexKey.MAX_LENGTH + 1));
+  }
+
+  @Test
+  void skipRejectsDecodedLengthOverMax() {
+    // Wire form that key() cannot produce: MAX_LENGTH + 1 plain bytes then EOF.
+    var buffer = ByteBuffer.allocate(IndexKey.MAX_LENGTH + 2);
+    for (int i = 0; i < IndexKey.MAX_LENGTH + 1; i++) {
+      buffer.put((byte) 'a');
+    }
+    buffer.put(EOF).flip();
+
+    assertThatIllegalArgumentException()
+        .isThrownBy(() -> skip(buffer.duplicate()))

Review Comment:
   Should we also do this test for `deserializeKey()`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [NoSQL] Count decoded length in IndexKey.skip [polaris]

Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy merged PR #5636:
URL: https://github.com/apache/polaris/pull/5636


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [NoSQL] Count decoded length in IndexKey.skip [polaris]

Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #5636:
URL: https://github.com/apache/polaris/pull/5636#discussion_r4130521882


##########
persistence/nosql/persistence/api/src/test/java/org/apache/polaris/persistence/nosql/api/index/TestIndexKey.java:
##########
@@ -80,6 +82,41 @@ static Stream<String> keyTooLong() {
         STRING_100 + STRING_100 + STRING_100 + STRING_100 + STRING_100 + STRING_100);
   }
 
+  /**
+   * Keys whose decoded length is under {@link IndexKey#MAX_LENGTH} but that expand when escaped
+   * must still be skippable. {@code skip} used to count two units per escaped byte and reject these
+   * keys after they had already been written.
+   */
+  @ParameterizedTest
+  @ValueSource(chars = {'\u0001', '\u0002'})
+  void skipAcceptsKeysThatExpandWhenEscaped(char escapedChar) {
+    // Decoded length 251: under MAX_LENGTH, but 251 escape pairs used to push skip's counter past
+    // 500 (2 * 251) and throw while deserializeKey still succeeded.
+    var indexKey = key(String.valueOf(escapedChar).repeat(251));
+    var buffer = ByteBuffer.allocate(indexKey.serializedSize());
+    indexKey.serialize(buffer).flip();
+
+    assertThat(deserializeKey(buffer.duplicate())).isEqualTo(indexKey);
+    var skipped = buffer.duplicate();
+    skip(skipped);
+    assertThat(skipped.remaining()).isZero();
+  }
+
+  @Test
+  void skipAcceptsMixedEscapedKeyNearLimit() {
+    // 250 escaped bytes + one ordinary byte: decoded length 251. The old skip counter would reach
+    // 501 (2 * 250 + 1) and throw.
+    var raw = "\u0001".repeat(250) + "a";

Review Comment:
   Yea, or maybe just mirror/update the existing `keyLengthGood`, `keyTooLong` test functions (add another assertion for `skip` in those). Could also enhance `keySerializationRoundTrip` and add calls to `skip` and compare the skipped count with the change to `ByteBuffer` positions.
   Adding another parameter to `static keyTooLong()` wouldn't hurt.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org