You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "uros-b (via GitHub)" <gi...@apache.org> on 2026/07/28 07:58:36 UTC
[PR] Core: Include CRC values in DV deserialization error message [iceberg]
uros-b opened a new pull request, #17393:
URL: https://github.com/apache/iceberg/pull/17393
"When a deletion vector fails CRC validation, the error message was a static "Invalid CRC". This includes the actual and expected CRC values to make the failure actionable, matching the sibling validation messages in the same class (`Invalid magic number/cardinality/bitmap data length: %s, expected %s`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #17393:
URL: https://github.com/apache/iceberg/pull/17393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #17393:
URL: https://github.com/apache/iceberg/pull/17393#issuecomment-5149889200
cc @huaxingao to PTAL ^^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #17393:
URL: https://github.com/apache/iceberg/pull/17393#issuecomment-5265289609
This PR should be good to go ^^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #17393:
URL: https://github.com/apache/iceberg/pull/17393#issuecomment-5188802059
cc @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #17393:
URL: https://github.com/apache/iceberg/pull/17393#discussion_r3691481453
##########
core/src/main/java/org/apache/iceberg/deletes/BitmapPositionDeleteIndex.java:
##########
@@ -150,7 +150,8 @@ public static PositionDeleteIndex deserialize(byte[] bytes, DeleteFile deleteFil
int crc = computeChecksum(bytes, bitmapDataLength);
int crcOffset = LENGTH_SIZE_BYTES + bitmapDataLength;
int expectedCrc = buffer.getInt(crcOffset);
- Preconditions.checkArgument(crc == expectedCrc, "Invalid CRC");
+ Preconditions.checkArgument(
+ crc == expectedCrc, "Invalid CRC: %s, expected %s", crc, expectedCrc);
Review Comment:
Thank you @nastra! addressed all in latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17393:
URL: https://github.com/apache/iceberg/pull/17393#discussion_r3681016142
##########
core/src/main/java/org/apache/iceberg/deletes/BitmapPositionDeleteIndex.java:
##########
@@ -150,7 +150,8 @@ public static PositionDeleteIndex deserialize(byte[] bytes, DeleteFile deleteFil
int crc = computeChecksum(bytes, bitmapDataLength);
int crcOffset = LENGTH_SIZE_BYTES + bitmapDataLength;
int expectedCrc = buffer.getInt(crcOffset);
- Preconditions.checkArgument(crc == expectedCrc, "Invalid CRC");
+ Preconditions.checkArgument(
+ crc == expectedCrc, "Invalid CRC: %s, expected %s", crc, expectedCrc);
Review Comment:
I guess the question is how actionable resulting error msg becomes. Maybe the error msg should also include the delete file location. Also what would a typical CRC tool spit out for a file? Would it be the same number or a hex string? In the latter case it might be worth to print the hex string representation of the two CRCs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17393:
URL: https://github.com/apache/iceberg/pull/17393#discussion_r3681018630
##########
core/src/main/java/org/apache/iceberg/deletes/BitmapPositionDeleteIndex.java:
##########
@@ -150,7 +150,8 @@ public static PositionDeleteIndex deserialize(byte[] bytes, DeleteFile deleteFil
int crc = computeChecksum(bytes, bitmapDataLength);
int crcOffset = LENGTH_SIZE_BYTES + bitmapDataLength;
int expectedCrc = buffer.getInt(crcOffset);
- Preconditions.checkArgument(crc == expectedCrc, "Invalid CRC");
+ Preconditions.checkArgument(
+ crc == expectedCrc, "Invalid CRC: %s, expected %s", crc, expectedCrc);
Review Comment:
Also would be great to add a small test that exercises this path and asserts on the resulting error msg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17393:
URL: https://github.com/apache/iceberg/pull/17393#discussion_r3701896185
##########
core/src/test/java/org/apache/iceberg/deletes/TestBitmapPositionDeleteIndex.java:
##########
@@ -176,6 +177,24 @@ public void testAllContainerTypesIndexSerialization() throws Exception {
validate(index, "all-container-types-position-index.bin");
}
+ @Test
+ public void testDeserializeInvalidCrc() {
+ PositionDeleteIndex index = new BitmapPositionDeleteIndex();
+ index.delete(1L);
+ index.delete(2L);
+ byte[] bytes = index.serialize().array();
+
+ // corrupt the last CRC byte so the computed checksum no longer matches
+ bytes[bytes.length - 1] ^= 0x01;
+
+ DeleteFile dv = mockDV(bytes.length, index.cardinality());
+ Mockito.when(dv.location()).thenReturn("s3://bucket/dv.puffin");
+
+ assertThatThrownBy(() -> PositionDeleteIndex.deserialize(bytes, dv))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("Invalid CRC for deletion vector s3://bucket/dv.puffin");
Review Comment:
the CRC should be deterministic, can we include it in the message and have only a `hasMessage(..)` check instead of just checking part of the message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #17393:
URL: https://github.com/apache/iceberg/pull/17393#discussion_r3702969155
##########
core/src/test/java/org/apache/iceberg/deletes/TestBitmapPositionDeleteIndex.java:
##########
@@ -176,6 +177,24 @@ public void testAllContainerTypesIndexSerialization() throws Exception {
validate(index, "all-container-types-position-index.bin");
}
+ @Test
+ public void testDeserializeInvalidCrc() {
+ PositionDeleteIndex index = new BitmapPositionDeleteIndex();
+ index.delete(1L);
+ index.delete(2L);
+ byte[] bytes = index.serialize().array();
+
+ // corrupt the last CRC byte so the computed checksum no longer matches
+ bytes[bytes.length - 1] ^= 0x01;
+
+ DeleteFile dv = mockDV(bytes.length, index.cardinality());
+ Mockito.when(dv.location()).thenReturn("s3://bucket/dv.puffin");
+
+ assertThatThrownBy(() -> PositionDeleteIndex.deserialize(bytes, dv))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("Invalid CRC for deletion vector s3://bucket/dv.puffin");
Review Comment:
Makes sense, added!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Include CRC values in DV deserialization error message [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #17393:
URL: https://github.com/apache/iceberg/pull/17393#issuecomment-5175801201
This one should be ready to merge @huaxingao @natra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org