Re: [I] Dictionary IDs Arrow IPC [arrow-rs]

["thinkharderdev (via GitHub)" <gi...@apache.org>]

thinkharderdev commented on issue #1206:
URL: https://github.com/apache/arrow-rs/issues/1206#issuecomment-1992172886

   > Another alternative is to remove `dict_id` entirely from `Field` and only use it while doing IPC serialization
   > 
   > This appears to be what @jorgecarleitao has done in arrow2
   > 
   > https://github.com/jorgecarleitao/arrow2/search?q=dict_id https://github.com/jorgecarleitao/arrow2/blob/main/src/datatypes/field.rs#L12-L21
   
   We're currently having a lot of issues with `dict_id` in arrow flight. There are a couple of different issues which I think @alamb suggestion above would alleviate:
   1. Ensuring that you assign a unique `dict_id` all the time ends up creating a lot of boilerplate since you need some way to track the current `dict_id`s and ensure that you always preserve the `dict_id` once it is assigned.
   2. `FlightDataEncoder` is inconsistent in where it gets dictionary ids from. When it sends the `schema` it will of course send the `dict_id` defined in the schema field. But when serializing the array it will send the `dict_id` from the array's datatype. Theoretically these should always be the same but since `PartialEq` on `Field` ignores the `dict_id` this is hard to enforce in practice. 
   3. To the last point, it is not even always possible to explicitly assign a `dict_id` when building an array with nested dictionary fields. Until very recently, using `GenericListBuilder` would just default the `dict_id` on it's `item` field. 
   
   
   That is all to say I'd like to work on making `dict_id` an internal implementation detail of the serialization and completely ignore the field's `dict_id` for that purpose. Then we can deprecate `Field::dict_id` and then eventually remove the `dict_id` field from `Field` entirely. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org