[Java] IPC stream write with re-stated dictionaries

[Joris Peeters <jo...@gmail.com>]

Hello,

For my use case I'm sending an Arrow IPC-stream from a server to a client,
with some columns being dictionary-encoded. Dictionary-encoding happens on
the fly, though, so the full dictionary isn't known yet at the beginning of
the stream, but rather is computed for every batch, and DictionaryBatches
are to be emitted prior to every RecordBatch.

However, unless I am mistaken, this is not currently supported in the
ArrowStreamWriter. The dictionary provider is passed in at construction
time, the dicts are emitted once, and there is no hook for re-emitting
these.

I've locally hacked around this by basically copy-pasting ArrowStreamWriter
and extending it with a   `public void writeBatch(DictionaryProvider
provider)` method, that re-emits the dictionaries prior to emitting the
record batches.

However, I'd of course much prefer if the provided ArrowStreamWriter
supported this. If people agree that it's missing (i.e. maybe I'm
overlooking something obvious) and that it would be useful to have, then
I'm happy to contribute it myself (not necessarily by using the
aforementioned `writeBatch(provider)` approach, but seems reasonable).

Cheers,
-J

Re: [Java] IPC stream write with re-stated dictionaries

[Joris Peeters <jo...@gmail.com>]

Cheers. Made and self-assigned
https://issues.apache.org/jira/browse/ARROW-11869.

On Fri, Mar 5, 2021 at 1:44 AM Micah Kornfield <em...@gmail.com>
wrote:

> Hi Joris,
> I do believe this is missing.  I believe we worked around this for testing
> by directly writing dictionary batches to the stream [1].
>
> Thanks,
> Micah
>
> [1]
>
> https://github.com/apache/arrow/blob/master/java/vector/src/test/java/org/apache/arrow/vector/ipc/TestArrowReaderWriter.java#L614
>
> On Thu, Mar 4, 2021 at 4:06 AM Joris Peeters <jo...@gmail.com>
> wrote:
>
> > Hello,
> >
> > For my use case I'm sending an Arrow IPC-stream from a server to a
> client,
> > with some columns being dictionary-encoded. Dictionary-encoding happens
> on
> > the fly, though, so the full dictionary isn't known yet at the beginning
> of
> > the stream, but rather is computed for every batch, and DictionaryBatches
> > are to be emitted prior to every RecordBatch.
> >
> > However, unless I am mistaken, this is not currently supported in the
> > ArrowStreamWriter. The dictionary provider is passed in at construction
> > time, the dicts are emitted once, and there is no hook for re-emitting
> > these.
> >
> > I've locally hacked around this by basically copy-pasting
> ArrowStreamWriter
> > and extending it with a   `public void writeBatch(DictionaryProvider
> > provider)` method, that re-emits the dictionaries prior to emitting the
> > record batches.
> >
> > However, I'd of course much prefer if the provided ArrowStreamWriter
> > supported this. If people agree that it's missing (i.e. maybe I'm
> > overlooking something obvious) and that it would be useful to have, then
> > I'm happy to contribute it myself (not necessarily by using the
> > aforementioned `writeBatch(provider)` approach, but seems reasonable).
> >
> > Cheers,
> > -J
> >
>

Re: [Java] IPC stream write with re-stated dictionaries

[Micah Kornfield <em...@gmail.com>]

Hi Joris,
I do believe this is missing.  I believe we worked around this for testing
by directly writing dictionary batches to the stream [1].

Thanks,
Micah

[1]
https://github.com/apache/arrow/blob/master/java/vector/src/test/java/org/apache/arrow/vector/ipc/TestArrowReaderWriter.java#L614

On Thu, Mar 4, 2021 at 4:06 AM Joris Peeters <jo...@gmail.com>
wrote:

> Hello,
>
> For my use case I'm sending an Arrow IPC-stream from a server to a client,
> with some columns being dictionary-encoded. Dictionary-encoding happens on
> the fly, though, so the full dictionary isn't known yet at the beginning of
> the stream, but rather is computed for every batch, and DictionaryBatches
> are to be emitted prior to every RecordBatch.
>
> However, unless I am mistaken, this is not currently supported in the
> ArrowStreamWriter. The dictionary provider is passed in at construction
> time, the dicts are emitted once, and there is no hook for re-emitting
> these.
>
> I've locally hacked around this by basically copy-pasting ArrowStreamWriter
> and extending it with a   `public void writeBatch(DictionaryProvider
> provider)` method, that re-emits the dictionaries prior to emitting the
> record batches.
>
> However, I'd of course much prefer if the provided ArrowStreamWriter
> supported this. If people agree that it's missing (i.e. maybe I'm
> overlooking something obvious) and that it would be useful to have, then
> I'm happy to contribute it myself (not necessarily by using the
> aforementioned `writeBatch(provider)` approach, but seems reasonable).
>
> Cheers,
> -J
>