[PR] fix(variant): write parquet field ids in shredded physical schema [paimon-cpp]

["SteNicholas (via GitHub)" <gi...@apache.org>]

SteNicholas opened a new pull request, #379:
URL: https://github.com/apache/paimon-cpp/pull/379

   ### Purpose
   
   Linked issue: close #344
   
   The generated physical schema of a shredded Variant column carried no `paimon.id`, so the parquet
   writer emitted no field id for `metadata`, `value`, `typed_value` or any of their descendants. Java
   Paimon converts the whole Variant group with its generic `ParquetSchemaConverter`, which
   dereferences the field id of every converted field, so reading such a file threw a
   `NullPointerException` before the first row. Configured, inferred and adaptive shredding share this
   schema construction, so all three produced files that only paimon-cpp could read.
   
   This numbers the generated fields the way the Java `PaimonShreddingUtils` does: each level is
   numbered from 0 in insertion order, an object field keeps the id from the shredding schema or its
   position when it has none, and an array element takes the id `ParquetSchemaConverter` generates for
   it (`SpecialFields.getArrayElementFieldId`).
   
   ### Tests
   
   - `VariantShreddingTest.ShreddingSchemaFieldIds`
   - `VariantShreddingTest.ShreddingSchemaRejectsInvalidFieldIds`
   - `VariantShreddingTest.ShreddingArraySchemaFieldIds`
   - `VariantParquetTest.ShreddedWriteAndReadRoundTrip`, extended to assert the raw parquet footer
     field ids of the whole shredded subtree under the configured, per-file and adaptive shredding
     modes
   - `SpecialFieldsTest.TestGetArrayElementFieldId`, pinning the generated element id against the
     values the Java `SpecialFields` produces
   
   ### API and Format
   
   No public API change. The storage format of shredded Variant columns changes: the generated
   physical fields now carry parquet field ids, matching what the Java writer produces. Files written
   before this change remain readable by paimon-cpp.
   
   ### Documentation
   
   No, this is a bug fix and introduces no new feature.
   
   ### Generative AI tooling
   
   Generated-by: Claude Code (Claude Opus 5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(variant): write parquet field ids in shredded physical schema [paimon-cpp]

["lxy-9602 (via GitHub)" <gi...@apache.org>]

lxy-9602 commented on PR #379:
URL: https://github.com/apache/paimon-cpp/pull/379#issuecomment-5759271363

   Thanks to @SteNicholas  for the fix. @suxiaogang223 , could you please check again whether the issue is resolved after this PR is merged? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(variant): write parquet field ids in shredded physical schema [paimon-cpp]

["lxy-9602 (via GitHub)" <gi...@apache.org>]

lxy-9602 merged PR #379:
URL: https://github.com/apache/paimon-cpp/pull/379


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org