You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Neuw84 (via GitHub)" <gi...@apache.org> on 2026/10/23 05:47:04 UTC
[PR] Core: order UUID bounds as unsigned so equality deletes are not dropped (#18619) [iceberg]
Neuw84 opened a new pull request, #18622:
URL: https://github.com/apache/iceberg/pull/18622
Closes #18619.
## Problem
An equality delete keyed on a `uuid` column (most visibly a composite key like `(uuid, int)`) is never attached to any `FileScanTask.deletes()`, so none of its rows are deleted. The loaded delete set is correct; the delete file simply never reaches the reader because scan planning decides it cannot apply.
Root cause is the UUID comparator. `Comparators` maps `UUIDType` to `Comparator.naturalOrder()`, i.e. `java.util.UUID.compareTo`, which compares the two halves as **signed** longs:
```java
.put(Types.UUIDType.get(), Comparator.naturalOrder())
```
But UUIDs are serialized and bounded as **16-byte big-endian** values (spec Appendix B/D), and Parquet computes a data file's min/max bounds over those bytes in **unsigned** order. So a data file's stored `[lower, upper]` uuid bounds form an unsigned interval.
When that interval straddles the signed boundary — the most significant long of either bound is negative as a signed `long`, i.e. the leading byte is `>= 0x80`, which is roughly half of all random UUIDs — the signed comparator sees `lower > upper` and treats the interval as inverted. In `DeleteFileIndex.canContainEqDeletesForFile` → `rangesOverlap`, the range check then concludes the delete cannot overlap the data file and the delete is dropped. With a composite `(uuid, int)` key the false per-field result ANDs into the whole check, so the delete is excluded reliably. Single-column `(int)` / `(string)` controls work because their comparators match their byte serialization.
## Fix
Order `UUIDType` with an unsigned 128-bit comparator — `Long.compareUnsigned` on the most significant bits, then the least — which matches the big-endian unsigned order the bounds are computed and serialized in. This fixes the mismatch for **every** uuid bounds comparison (delete planning, manifest pruning, `ParquetMetricsRowGroupFilter`, residual evaluation), not just this one call site.
It is a change to UUID sort order, but toward the spec-mandated big-endian unsigned order; the only behavior that relied on the previous signed order was relying on the bug.
## Tests
- `TestComparators.testUuid`: adds pairs that cross the signed/unsigned boundary (e.g. `7fff.. < 8000..`, `00.. < ff..`, and an MSB tie that falls through to the least significant bits). These **fail** under the old signed comparator and pass with the fix. The pre-existing pair is kept and still holds (both of its UUIDs sort the same way signed or unsigned, which is why the bug went unnoticed).
- `DeleteFileIndexTestBase.testEqualityDeleteWithUuidKeyAttachesToDataFile`: an end-to-end scan-planning test. A data file whose uuid bounds `[00..01, ff..fe]` straddle the boundary, and a `(uuid, int)` equality delete whose uuid key `c0..` lies inside the range only under unsigned ordering, must be attached (`forDataFile(...).hasSize(1)`). Fails under the old comparator (the delete is dropped), passes with the fix. Runs under every parameterized format version in the base, so it covers the local and REST scan-planning subclasses.
## Build / test note
I could not run the Gradle suite locally — this environment only has JDK 25 and the Iceberg build requires JDK 17 or 21 (`This build must be run with JDK 17 or 21`). The change is small and both new tests pin the exact ordering contract; CI will exercise them under a supported JDK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: order UUID bounds as unsigned so equality deletes are not dropped (#18619) [iceberg]
Posted by "Neuw84 (via GitHub)" <gi...@apache.org>.
Neuw84 commented on PR #18622:
URL: https://github.com/apache/iceberg/pull/18622#issuecomment-6368032162
@JeonDaehong you're right, and thanks for the careful reproduction — the two orders problem and the `WHERE u = <uuid>` regression are real, and I should not have changed `Comparators` in isolation.
To restate your finding so it's on the record:
- My change flips only `Comparators.forType(UUID)` to unsigned. That's the write/bounds side: `PartitionSummary` builds manifest-list partition bounds through `Comparators.forType`, so they now come out in unsigned order.
- But the read side — `ManifestEvaluator`, `InclusiveMetricsEvaluator`, `ParquetMetricsRowGroupFilter` — compares through `lit.comparator()`, and `Literals.UUIDLiteral` is still a `ComparableLiteral` using signed `UUID.compareTo`. (Note `FixedLiteral`/`BinaryLiteral` already use `Comparators.unsignedBytes()`; UUID is the one 16-byte type left on signed order.)
- So the two sides disagree, and your `identity(u)` scan shows the consequence: an unpartitioned file with bounds `[00…0001, ff…fffe]` and `equal("u", c0…)` now plans 0 files on this PR where it planned 1 on main. That is a regression I introduced, in exactly the plain `WHERE u = …` case from #14216. I confirmed the mechanism matches what you described.
So flipping `Comparators` alone is not a safe partial fix — it has to be the literals + evaluators aligned together, which is #14500.
And that's the heart of it: the correct fix here **is** #14500. That PR already lines up the literals, the evaluators and the write path, and — crucially — it carries the backward-compatibility logic that a bare flip to unsigned cannot: metadata written in signed order by Iceberg Java ≤1.11 must still be read, which is why bodduv's approach evaluates the predicate twice (RFC-compliant unsigned **and** signed, filtering in if either matches) and returns `ROWS_MIGHT_NOT_MATCH` for UUID in `StrictMetricsEvaluator`. My one-line comparator change has none of that, so even "finish it properly in this PR" would mean re-implementing #14500 — and #14500 was explicitly held pending a dev-list discussion and a community-sync decision on how UUID ordering should be defined in the spec, since it is a cross-version behavioral change.
I don't think this PR should race ahead of that. Two ways forward, and I'd like your (and ideally @bodduv's) read:
1. **Fold the equality-delete fix into #14500.** The `DeleteFileIndex` drop you confirmed as correct here is the same root cause as the `WHERE u = …` drop #14500 targets — one unsigned-ordering fix across literals + evaluators + delete planning covers both, with the signed-metadata read compatibility handled once. bodduv offered on Oct 8 to split out the read-path changes first; the equality-delete case is a natural part of that read-path slice, and I'm happy to contribute a test like `testEqualityDeleteWithUuidKeyAttachesToDataFile` there.
2. **Keep #18622 only if it can be scoped to not touch `Comparators.forType`** — but I don't think that scoping exists, because `DeleteFileIndex.canContainEqDeletesForFile` → `rangesOverlap` goes through that very comparator. Any fix to the delete drop changes UUID bounds comparison and reintroduces the same two-orders split you found. So I lean strongly toward (1).
My preference is to close this in favor of #14500 (or to retarget it as the read-path slice bodduv described), rather than ship a partial flip that trades one correctness bug for another. I'll hold the PR as-is pending that decision. @bodduv — if you're splitting out the read path, would an equality-delete regression test be welcome in that slice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: order UUID bounds as unsigned so equality deletes are not dropped (#18619) [iceberg]
Posted by "JeonDaehong (via GitHub) via issues" <is...@iceberg.apache.org>.
JeonDaehong commented on code in PR #18622:
URL: https://github.com/apache/iceberg/pull/18622#discussion_r4577046204
##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -46,7 +47,7 @@ private Comparators() {}
.put(Types.TimestampNanoType.withZone(), Comparator.naturalOrder())
.put(Types.TimestampNanoType.withoutZone(), Comparator.naturalOrder())
.put(Types.StringType.get(), Comparators.charSequences())
- .put(Types.UUIDType.get(), Comparator.naturalOrder())
+ .put(Types.UUIDType.get(), Comparators.uuids())
Review Comment:
Agreed, the two paths need the same order before this can go in. I couldn't find a dev@ thread on UUID ordering yet. If it helps, I can start one with the cases from this PR: the manifest-list bounds and the metrics evaluators disagreeing on UUIDs whose high bit differs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: order UUID bounds as unsigned so equality deletes are not dropped (#18619) [iceberg]
Posted by "CTTY (via GitHub) via issues" <is...@iceberg.apache.org>.
CTTY commented on code in PR #18622:
URL: https://github.com/apache/iceberg/pull/18622#discussion_r4571642916
##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -46,7 +47,7 @@ private Comparators() {}
.put(Types.TimestampNanoType.withZone(), Comparator.naturalOrder())
.put(Types.TimestampNanoType.withoutZone(), Comparator.naturalOrder())
.put(Types.StringType.get(), Comparators.charSequences())
- .put(Types.UUIDType.get(), Comparator.naturalOrder())
+ .put(Types.UUIDType.get(), Comparators.uuids())
Review Comment:
I hit the same mismatch when testing something. UUIDs are compared through two paths, and this PR changes only one:
- (fixed by this PR) `Comparators.forType(UUIDType)` → now unsigned. Used on the **write** side (`PartitionSummary`, `ParquetMetrics`) and by `DeleteFileIndex`.
- `Literal.comparator()` (`Literals.UUIDLiteral` is a `ComparableLiteral`, i.e. `UUID.compareTo`) → still signed. Used by every evaluator: `ManifestEvaluator`, `InclusiveMetricsEvaluator`, `StrictMetricsEvaluator`, `ParquetMetricsRowGroupFilter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: order UUID bounds as unsigned so equality deletes are not dropped (#18619) [iceberg]
Posted by "JeonDaehong (via GitHub)" <gi...@apache.org>.
JeonDaehong commented on PR #18622:
URL: https://github.com/apache/iceberg/pull/18622#issuecomment-6378419989
Agree with (1).
One small correction to the restatement, so it doesn't confuse things later on #14500: the regression from this PR is the `identity(u)` case, where manifest pruning goes from 1 file to 0. The unpartitioned `[00…01, ff…fe]` file plans 0 on main as well. That's the existing #14216 bug, not something this PR introduced.
Moving the eq-delete test into bodduv's read-path slice sounds good to me. #18619 is the case users actually reported, so it's a useful regression test for that slice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: order UUID bounds as unsigned so equality deletes are not dropped (#18619) [iceberg]
Posted by "Neuw84 (via GitHub)" <gi...@apache.org>.
Neuw84 commented on PR #18622:
URL: https://github.com/apache/iceberg/pull/18622#issuecomment-6358391914
Related: #18621 — a sibling equality-delete correctness fix from the same native-scan work. Different root cause and disjoint code (that one is a reader buffer-aliasing bug for `fixed` keys in `parquet`; this one is an unsigned-UUID-ordering bug in scan planning, `api`/`core`), so the two are independent and can merge in any order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: order UUID bounds as unsigned so equality deletes are not dropped (#18619) [iceberg]
Posted by "JeonDaehong (via GitHub)" <gi...@apache.org>.
JeonDaehong commented on code in PR #18622:
URL: https://github.com/apache/iceberg/pull/18622#discussion_r4360131045
##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -46,7 +47,7 @@ private Comparators() {}
.put(Types.TimestampNanoType.withZone(), Comparator.naturalOrder())
.put(Types.TimestampNanoType.withoutZone(), Comparator.naturalOrder())
.put(Types.StringType.get(), Comparators.charSequences())
- .put(Types.UUIDType.get(), Comparator.naturalOrder())
+ .put(Types.UUIDType.get(), Comparators.uuids())
Review Comment:
This also changes `PartitionSummary`, which builds the manifest-list partition bounds with `Comparators.forType`. The read side compares them through `lit.comparator()`, which is still signed for UUID.
`identity(u)` table, two files with `u = 00000000-...-0001` and `u = c0000000-...-0000`, `newScan().filter(equal("u", x)).planFiles()`:
| filter | main | this PR |
|---|---|---|
| `u = 00...01` | 1 file | 0 files |
| `u = c0...00` | 1 file | 0 files |
The new summary is lower=`00...01`, upper=`c0...`. The signed literal compare sees `upper < value` for both lookups and skips the manifest. On main both sides are signed, so this works today.
##########
core/src/test/java/org/apache/iceberg/DeleteFileIndexTestBase.java:
##########
@@ -796,4 +797,87 @@ public void testInvalidDVSequenceNumber() {
.isInstanceOf(ValidationException.class)
.hasMessageContaining("must be greater than or equal to data file sequence number");
}
+
+ @TestTemplate
+ public void testEqualityDeleteWithUuidKeyAttachesToDataFile() {
Review Comment:
Could we also have a scan-level test with an `identity(u)` partition (or a `ManifestEvaluator` test)? This one only goes through `DeleteFileIndex`, so it wouldn't catch the manifest pruning change above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org