[PR] [core] Align lost and corrupt data-file recovery with Java [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi opened a new pull request, #1049:
URL: https://github.com/apache/paimon-rust/pull/1049

   ### Purpose
   
   Closes #1048.
   
   Honor Java's `scan.ignore-lost-files` and `scan.ignore-corrupt-files` in Paimon data-file reads, including the paths used by PyPaimon Native. Previously these options could not recover a missing or corrupt primary Parquet file; the lost-files option only supported ROW auxiliary-file fallback.
   
   ### Brief change log
   
   - Add a private file-read recovery policy shared by normal and local-range readers. Creation failures check actual file existence; batch failures require `scan.ignore-corrupt-files`.
   - Retain batches returned before a decode failure and continue subsequent files.
   - End a Data Evolution column group when an active file is actually skipped, following Java `DataEvolutionFileReader`. Keep ordinary row-count and alignment validation strict.
   - Apply the same recovery semantics to PK vector position and fallback reads, without weakening their overrun checks.
   - Propagate resource, configuration, fork-safety and unsupported-operation failures, including wrapped errors. Schema mapping, deletion vectors and external BLOB payload resolution remain outside file recovery.
   - Keep both options disabled by default. Reads never remove files.
   
   ### Tests
   
   - `cargo test --locked -p paimon --all-targets --features fulltext,vortex`: 4,253 passed, 6 ignored.
   - `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings`
   - `cargo fmt --all -- --check`
   - New regression cases cover append/Data Evolution/PK, missing versus corrupt files, disabled and opposite options, zero-column reads, stable row IDs, damaged update-column groups, external BLOB errors, resource limits and PK position reads.
   - A real Parquet file with an intact footer and damaged second row group verifies preservation of emitted batches and prefetch-resource release.
   
   ### API and Format
   
   Adds `CoreOptions::scan_ignore_corrupt_file()` for the existing Java option. No storage-format changes or Python-binding API changes.
   
   ### Documentation
   
   Option semantics and Java alignment are documented in the recovery helper. Related PyPaimon coverage will be submitted separately and requires this change to be merged before the Native CI job can pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Align lost and corrupt data-file recovery with Java [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed pull request #1049: [core] Align lost and corrupt data-file recovery with Java
URL: https://github.com/apache/paimon-rust/pull/1049


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org