[jira] [Assigned] (ARROW-11869) [Java] Support re-emitting dictionaries in ArrowStreamWriter

["Alessandro Molina (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-11869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alessandro Molina reassigned ARROW-11869:
-----------------------------------------

    Assignee:     (was: Joris Peeters)

> [Java] Support re-emitting dictionaries in ArrowStreamWriter
> ------------------------------------------------------------
>
>                 Key: ARROW-11869
>                 URL: https://issues.apache.org/jira/browse/ARROW-11869
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Java
>            Reporter: Joris Peeters
>            Priority: Minor
>
> The ArrowStreamWriter currently takes a DictionaryProvider at construction time and emits the used dicts once.
> However, the streaming format allows for the dictionaries to change between record batches. It would be useful to support this mechanism. It can be worked around in various ways (e.g. manually re-emitting DictionaryBatches between calling writeBatch), but this isn't very pleasant.
> We'd somehow have to reconcile this with the abstract ArrowWriter parent and the ArrowFileWriter sibling. In the latter, for example, this mechanism is not supported.
> An example solution (but perhaps we can do better) might be to add a virtual `writeBatch(Provider provider)` method, that is UnsupportedOperationException in ArrowFileWriter, and re-emits the used dicts in ArrowStreamWriter.
> In the present context just looking at dictionary replacement, not dictionary delta's.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)