You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/09/22 15:59:37 UTC
[PR] [core] Rebuild file index over the correct column after schema evolution [paimon]
jackylee-ch opened a new pull request, #10122:
URL: https://github.com/apache/paimon/pull/10122
### Purpose
`FileIndexProcessor` (the `rewrite_file_index` procedure) rebuilds a data file's file index. It computes `projectedIndexCols` as the index columns' positions in the **file** schema and builds the writer with `fileSchema.project(projectedIndexCols)`. But the reader used `ReadBuilder.withProjection(projectedIndexCols)`, which `ReadBuilderImpl` resolves against the **current table** schema.
For a file whose `schemaId` differs from current, the spaces diverge once columns shift: `[k, a, v]` (bloom on `v`) → drop `a`, add `w` leaves `v` at file-index 2 while current-index 2 is now `w`. The reader then reads `w` (null-filled for the old file), so the bloom is rebuilt over nulls and a later `v = 100` query silently prunes the file.
Fix: read with `withReadType(fileSchema.project(projectedIndexCols))` — the writer's own file-schema projection — so reader and writer share the file's column space. Current-schema files are unchanged.
### Tests
`testRebuildsIndexOnCorrectColumnAfterColumnDropAndAdd`: bloom on `v` in `[k, a, v]`, write `v=100`, drop `a`, add `w`, rebuild, assert the `v` bloom still reports `100` present. Fails on master (wrong column → `100` absent); passes here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rebuild file index over the correct column after schema evolution [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10122:
URL: https://github.com/apache/paimon/pull/10122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rebuild file index over the correct column after schema evolution [paimon]
Posted by "Akash3121 (via GitHub)" <gi...@apache.org>.
Akash3121 commented on code in PR #10122:
URL: https://github.com/apache/paimon/pull/10122#discussion_r4078259035
##########
paimon-core/src/test/java/org/apache/paimon/index/FileIndexProcessorTest.java:
##########
@@ -174,4 +176,79 @@ public void testProcessReadsTheSchemasOfTheTableBranch() throws Exception {
DataFileMeta processed = processor.process(entry.partition(), entry.bucket(), entry);
assertThat(processed.extraFiles()).isNotEmpty();
}
+
+ @Test
+ public void testRebuildsIndexOnCorrectColumnAfterColumnDropAndAdd() throws Exception {
+ LocalFileIO fileIO = LocalFileIO.create();
+ Path warehouse = new Path(tempDir.toString());
+ Map<String, String> options = new HashMap<>();
+ options.put(CoreOptions.BUCKET.key(), "1");
+ options.put(CoreOptions.FILE_FORMAT.key(), "parquet");
+ options.put(CoreOptions.FILE_INDEX + ".bloom-filter.columns", "v");
Review Comment:
non blocking
Could this test use ` file-index.bitmap.columns` instead of a Bloom filter? A Bloom-filter positive is probabilistic, so `remain() == true ` does not strictly prove that `100` was indexed from `v` ; a false positive could allow the old wrong-column behavior to pass. The bitmap index returns an exact empty result for an absent value, making this regression deterministic while preserving the same schema-evolution scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org