[PR] [core] Pass data file context to FileIndexer writers [paimon]

["wangyong9999 (via GitHub)" <gi...@apache.org>]

wangyong9999 opened a new pull request, #10315:
URL: https://github.com/apache/paimon/pull/10315

   ### Purpose
   
   Close #10314.
   
   A `FileIndexWriter` receives every row of a data file in file order, but `FileIndexer#createWriter()` gives it no information about which data file it indexes. Plugins whose output references the data file, such as an external `key -> (data file, row position)` mapping, have to rely on fragile workarounds.
   
   This change passes a `FileIndexWriterContext` with the data file path and schema id to file index writers:
   
   - Add `FileIndexWriterContext` and a default method `FileIndexer#createWriter(FileIndexWriterContext)` that falls back to `createWriter()`, so built-in indexes and existing plugins are unaffected.
   - `DataFileIndexWriter` takes the context and passes it to top-level and nested map index writers.
   - Every call site provides the context of the data file being indexed: `RowDataFileWriter`, `KeyValueDataFileWriter`, `KeyValueClusteringFileWriter`, and `FileIndexProcessor`, which passes the existing file and its original schema id when rewriting indexes.
   
   No storage format change.
   
   ### Tests
   
   - `DataFileIndexWriterTest`: an append table rolled into several files across a schema change, and a primary-key table; every data file has exactly one index writer, which receives the file path, the file's schema id and the indexed values in file order.
   - `FileIndexProcessorTest`: rewriting the index of an existing file passes that file and its original schema id.
   - `mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest='*FileIndex*,*FileWriter*Test,AppendOnlyWriterTest,KeyValueFileReadWriteTest,AppendOnlySimpleTableTest' test`: 300 tests passed. The three new tests fail without this change.
   - Java Spotless, Checkstyle and `git diff --check` passed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Pass data file context to FileIndexer writers [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on PR #10315:
URL: https://github.com/apache/paimon/pull/10315#issuecomment-5934638698

   Thanks for the contribution. I reviewed the append/primary-key/clustering writers, nested map indexes and index rewriting with historical schemas, and ran `DataFileIndexWriterTest` and `FileIndexProcessorTest` on JDK 8 with the normal Maven checks enabled: all 9 tests passed. The default method preserves existing `FileIndexer` implementations, and I did not identify an implementation regression.
   
   The standalone change does not yet deliver the end-to-end lookup benefit described in the motivation. Every production index implementation still uses the default fallback; the only implementation consuming `FileIndexWriterContext` is the recording test plugin. There is no production key-to-file/row-position index, query consumer, or write → commit → indexed lookup test in this PR. Issue #10314 also explicitly scopes this as a building block rather than the lookup implementation.
   
   Closing this standalone API extension for now. Please bring the context API back together with a concrete index consumer and tests demonstrating the complete lookup behavior, including compaction/rewrite and failed or retried commits. That will let us review the API against its actual lifecycle requirements and verify the user-visible benefit.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Pass data file context to FileIndexer writers [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed pull request #10315: [core] Pass data file context to FileIndexer writers
URL: https://github.com/apache/paimon/pull/10315


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org