You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/06/26 18:20:08 UTC
[PR] [core] Add row sidecars for data evolution sparse reads [paimon]
JingsongLi opened a new pull request, #8365:
URL: https://github.com/apache/paimon/pull/8365
## Summary
Add opt-in row-store sidecar files for data evolution tables so sparse row-id reads can use `.row` files while normal data files keep their configured columnar format. The implementation covers both Java and Python read/write paths and keeps dedicated blob/vector writers sidecar-free.
## Changes
- Add explicit table options for row sidecar generation and sparse-read thresholds.
- Reuse the data-file auxiliary writer path to write file indexes and row sidecars from `RowDataFileWriter`.
- Route sparse row-id reads to a single `.row` sidecar when the selected row count and ratio are below the configured limits.
- Keep dedicated format writers from generating `.row` sidecars, and handle bundle writes through the same row path so sidecars and sequence ranges stay aligned.
- Mirror row sidecar write/read support in Python, including row-format reads with schema-evolution missing-field handling.
- Update generated option docs and add focused Java/Python tests.
## Testing
- [x] `mvn -pl paimon-api,paimon-core -am -Pfast-build -DskipTests compile`
- [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=DataFileIndexWriterTest,DataEvolutionSplitReadTest,RollingFileWriterTest,DedicatedFormatRollingFileWriterTest,DataEvolutionTableTest#testRowSidecarDisabledByDefault+testRowSidecarEnabledByTableOption test`
- [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=RollingFileWriterTest,DataEvolutionSplitReadTest test`
- [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=AppendOnlySimpleTableTest#testBatchRecordsWrite,PrimaryKeySimpleTableTest#testBatchRecordsWrite test`
- [x] `python3 -m py_compile paimon-python/pypaimon/common/options/core_options.py paimon-python/pypaimon/read/split_read.py paimon-python/pypaimon/tests/data_evolution_formats_test.py`
- [x] `python3 -m pytest paimon-python/pypaimon/tests/data_evolution_formats_test.py -k row_sidecar -q`
- [x] `git diff --check`
## Notes
Row sidecar generation is disabled by default through `data-evolution.row-sidecar.enabled=false`. The default read-side guardrails are `data-evolution.row-sidecar.max-selected-rows=4096` and `data-evolution.row-sidecar.max-selection-ratio=0.05`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add row sidecars for data evolution sparse reads [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8365:
URL: https://github.com/apache/paimon/pull/8365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org