You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Rich-T-kid (via GitHub)" <gi...@apache.org> on 2026/10/21 19:28:20 UTC

[PR] fix(parquet): honor requested dictionary value type [arrow-rs]

Rich-T-kid opened a new pull request, #11780:
URL: https://github.com/apache/arrow-rs/pull/11780

   # Which issue does this PR close?
   
   - Closes #11748
   - Part of apache/datafusion#24111
   
   # Rationale for this change
   
   DataFusion's `binary_as_string` and RLE dictionary options need an unannotated Parquet `BYTE_ARRAY` to map directly to `Dictionary<K, Utf8>`. The reader currently chooses the dictionary value type from the Parquet annotation, so it still builds a `Binary` child when the supplied Arrow schema requests `Utf8`. DataFusion then has to expand and rebuild the dictionary (apache/datafusion#26530).
   
   # What changes are included in this PR?
   
   Decode dictionary values using the value type requested by the supplied Arrow schema. This produces `Dictionary<K, Utf8>` directly, preserves the Parquet keys and shared values across batches, and validates UTF-8 while decoding.
   
   # Are these changes tested?
   
   Yes. Tests cover binary dictionaries requested as UTF-8 dictionaries across multiple batches and invalid UTF-8 input.
   
   # Are there any significant user-facing changes?
   
   Supplied dictionary schemas now produce columns with the requested value type. There are no public API changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(parquet): honor requested dictionary value type [arrow-rs]

Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid closed pull request #11780: fix(parquet): honor requested dictionary value type
URL: https://github.com/apache/arrow-rs/pull/11780


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org