You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/10/23 15:02:15 UTC

[GitHub] [arrow] alamb commented on pull request #8402: ARROW-8426: [Rust] [Parquet] - Add more support for converting Dicts

alamb commented on pull request #8402:
URL: https://github.com/apache/arrow/pull/8402#issuecomment-715396915


   > @vertexclique @nevi-me I'm feeling stuck on converting primitive dictionaries...
   > 
   > I have [a solution that works for one key/value type](https://github.com/apache/arrow/pull/8402/commits/4b59fc952336bee757cf27ae596b56edbccf099a), but I [tried to expand that to all the types](https://github.com/apache/arrow/pull/8402/commits/79b78d97f5457895f2e96a39dcc341e29a588058) and it involves listing out all the possible combinations (😱) and overflows the stack (😱😱😱).
   > 
   > I have tried to find a different abstraction, though, and the type checker doesn't like anything I've come up with. Do you have any suggestions?
   
   @carols10cents  -- one idea I had which might be less efficient at runtime but possibly be less complicated to implement, would be to use the arrow `cast` kernels here: https://github.com/apache/arrow/blob/master/rust/arrow/src/compute/kernels/cast.rs
   
   So rather than going directly from `ParquetType` to `DesiredArrowType` we could go from `ParquetType --> CanonicalArrowType` and then from `CanonicalArrowType` --> `DesiredArrowType`
   
   So for example, to generate a Dictionary<UInt8, Utf8> from a parquet column of `Utf8` you could always create `Dictionary<Uint64, Utf8>` and then use `cast` to go to the desired arrow type
   
   Does that make sense?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org