Re: [C++] LZ4 Compression formats

[Micah Kornfield <em...@gmail.com>]

>
> Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow
> 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe
> the meaning has changed between versions.


Sorry for the late reply.  yes, I think there have been a few rounds to
disambiguate this frmo 0.16.0 through 3.0.0

On Tue, Jun 22, 2021 at 12:45 PM Rares Vernica <rv...@gmail.com> wrote:

> Thanks for the pointer.
>
> Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow
> 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe
> the meaning has changed between versions.
>
> On Tue, Jun 22, 2021 at 12:33 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > It sounds like you are trying to use the buffer level compression
> described
> > by the specification? If so only LZ4_FRAME is currently allowed [1]
> >
> > [1] https://github.com/apache/arrow/blob/master/format/Message.fbs#L45
> >
> >
> > On Tue, Jun 22, 2021 at 12:28 PM Rares Vernica <rv...@gmail.com>
> wrote:
> >
> > > Hello,
> > >
> > > Using Arrow 3.0.0 I tried to compress a stream with LZ4 and got this
> > error
> > > message:
> > >
> > > NotImplemented: Streaming compression unsupported with LZ4 raw format.
> > Try
> > > using LZ4 frame format instead.
> > >
> > > Is it because LZ4 raw was not enabled when the .so was compiled or
> > actually
> > > not implemented?
> > >
> > > Is LZ4 frame the recommended format for compressing stream/IPC buffers?
> > >
> > > Thank you!
> > > Rares
> > >
> >
>

Re: [C++] LZ4 Compression formats

[Wes McKinney <we...@gmail.com>]

When the streaming compression interfaces were originally implemented
in 2018 [1], there was not a distinction between LZ4 "raw" compression
(which is Compression::LZ4) and LZ4 "frame" compression
(Compression::LZ4_FRAME). So in that patch while the LZ4 raw
compression method was being used for one-shot compression, the frame
format was being used for streaming.

Commingling these codec types proved problematic, so it was necessary
later (in 0.17.0 [2]) to distinguish the codecs so they can be invoked
separately for one-shot compression, while streaming compression is
only available for the frame format.

[1]: https://github.com/apache/arrow/commit/65f03168f20c9429768f64b761209365a99cbada
[2]: https://github.com/apache/arrow/commit/895f22009f945371fb18d00324cc5eb853dda51a

On Tue, Jul 6, 2021 at 5:08 AM Micah Kornfield <em...@gmail.com> wrote:
>
> >
> > Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow
> > 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe
> > the meaning has changed between versions.
>
>
> Sorry for the late reply.  yes, I think there have been a few rounds to
> disambiguate this frmo 0.16.0 through 3.0.0
>
> On Tue, Jun 22, 2021 at 12:45 PM Rares Vernica <rv...@gmail.com> wrote:
>
> > Thanks for the pointer.
> >
> > Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow
> > 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe
> > the meaning has changed between versions.
> >
> > On Tue, Jun 22, 2021 at 12:33 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > It sounds like you are trying to use the buffer level compression
> > described
> > > by the specification? If so only LZ4_FRAME is currently allowed [1]
> > >
> > > [1] https://github.com/apache/arrow/blob/master/format/Message.fbs#L45
> > >
> > >
> > > On Tue, Jun 22, 2021 at 12:28 PM Rares Vernica <rv...@gmail.com>
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > Using Arrow 3.0.0 I tried to compress a stream with LZ4 and got this
> > > error
> > > > message:
> > > >
> > > > NotImplemented: Streaming compression unsupported with LZ4 raw format.
> > > Try
> > > > using LZ4 frame format instead.
> > > >
> > > > Is it because LZ4 raw was not enabled when the .so was compiled or
> > > actually
> > > > not implemented?
> > > >
> > > > Is LZ4 frame the recommended format for compressing stream/IPC buffers?
> > > >
> > > > Thank you!
> > > > Rares
> > > >
> > >
> >