You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Rares Vernica <rv...@gmail.com> on 2021/06/22 19:28:27 UTC
[C++] LZ4 Compression formats
Hello,
Using Arrow 3.0.0 I tried to compress a stream with LZ4 and got this error
message:
NotImplemented: Streaming compression unsupported with LZ4 raw format. Try
using LZ4 frame format instead.
Is it because LZ4 raw was not enabled when the .so was compiled or actually
not implemented?
Is LZ4 frame the recommended format for compressing stream/IPC buffers?
Thank you!
Rares
Re: [C++] LZ4 Compression formats
Posted by Wes McKinney <we...@gmail.com>.
When the streaming compression interfaces were originally implemented
in 2018 [1], there was not a distinction between LZ4 "raw" compression
(which is Compression::LZ4) and LZ4 "frame" compression
(Compression::LZ4_FRAME). So in that patch while the LZ4 raw
compression method was being used for one-shot compression, the frame
format was being used for streaming.
Commingling these codec types proved problematic, so it was necessary
later (in 0.17.0 [2]) to distinguish the codecs so they can be invoked
separately for one-shot compression, while streaming compression is
only available for the frame format.
[1]: https://github.com/apache/arrow/commit/65f03168f20c9429768f64b761209365a99cbada
[2]: https://github.com/apache/arrow/commit/895f22009f945371fb18d00324cc5eb853dda51a
On Tue, Jul 6, 2021 at 5:08 AM Micah Kornfield <em...@gmail.com> wrote:
>
> >
> > Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow
> > 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe
> > the meaning has changed between versions.
>
>
> Sorry for the late reply. yes, I think there have been a few rounds to
> disambiguate this frmo 0.16.0 through 3.0.0
>
> On Tue, Jun 22, 2021 at 12:45 PM Rares Vernica <rv...@gmail.com> wrote:
>
> > Thanks for the pointer.
> >
> > Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow
> > 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe
> > the meaning has changed between versions.
> >
> > On Tue, Jun 22, 2021 at 12:33 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > It sounds like you are trying to use the buffer level compression
> > described
> > > by the specification? If so only LZ4_FRAME is currently allowed [1]
> > >
> > > [1] https://github.com/apache/arrow/blob/master/format/Message.fbs#L45
> > >
> > >
> > > On Tue, Jun 22, 2021 at 12:28 PM Rares Vernica <rv...@gmail.com>
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > Using Arrow 3.0.0 I tried to compress a stream with LZ4 and got this
> > > error
> > > > message:
> > > >
> > > > NotImplemented: Streaming compression unsupported with LZ4 raw format.
> > > Try
> > > > using LZ4 frame format instead.
> > > >
> > > > Is it because LZ4 raw was not enabled when the .so was compiled or
> > > actually
> > > > not implemented?
> > > >
> > > > Is LZ4 frame the recommended format for compressing stream/IPC buffers?
> > > >
> > > > Thank you!
> > > > Rares
> > > >
> > >
> >
Re: [C++] LZ4 Compression formats
Posted by Micah Kornfield <em...@gmail.com>.
>
> Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow
> 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe
> the meaning has changed between versions.
Sorry for the late reply. yes, I think there have been a few rounds to
disambiguate this frmo 0.16.0 through 3.0.0
On Tue, Jun 22, 2021 at 12:45 PM Rares Vernica <rv...@gmail.com> wrote:
> Thanks for the pointer.
>
> Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow
> 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe
> the meaning has changed between versions.
>
> On Tue, Jun 22, 2021 at 12:33 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > It sounds like you are trying to use the buffer level compression
> described
> > by the specification? If so only LZ4_FRAME is currently allowed [1]
> >
> > [1] https://github.com/apache/arrow/blob/master/format/Message.fbs#L45
> >
> >
> > On Tue, Jun 22, 2021 at 12:28 PM Rares Vernica <rv...@gmail.com>
> wrote:
> >
> > > Hello,
> > >
> > > Using Arrow 3.0.0 I tried to compress a stream with LZ4 and got this
> > error
> > > message:
> > >
> > > NotImplemented: Streaming compression unsupported with LZ4 raw format.
> > Try
> > > using LZ4 frame format instead.
> > >
> > > Is it because LZ4 raw was not enabled when the .so was compiled or
> > actually
> > > not implemented?
> > >
> > > Is LZ4 frame the recommended format for compressing stream/IPC buffers?
> > >
> > > Thank you!
> > > Rares
> > >
> >
>
Re: [C++] LZ4 Compression formats
Posted by Rares Vernica <rv...@gmail.com>.
Thanks for the pointer.
Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow
0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe
the meaning has changed between versions.
On Tue, Jun 22, 2021 at 12:33 PM Micah Kornfield <em...@gmail.com>
wrote:
> It sounds like you are trying to use the buffer level compression described
> by the specification? If so only LZ4_FRAME is currently allowed [1]
>
> [1] https://github.com/apache/arrow/blob/master/format/Message.fbs#L45
>
>
> On Tue, Jun 22, 2021 at 12:28 PM Rares Vernica <rv...@gmail.com> wrote:
>
> > Hello,
> >
> > Using Arrow 3.0.0 I tried to compress a stream with LZ4 and got this
> error
> > message:
> >
> > NotImplemented: Streaming compression unsupported with LZ4 raw format.
> Try
> > using LZ4 frame format instead.
> >
> > Is it because LZ4 raw was not enabled when the .so was compiled or
> actually
> > not implemented?
> >
> > Is LZ4 frame the recommended format for compressing stream/IPC buffers?
> >
> > Thank you!
> > Rares
> >
>
Re: [C++] LZ4 Compression formats
Posted by Micah Kornfield <em...@gmail.com>.
It sounds like you are trying to use the buffer level compression described
by the specification? If so only LZ4_FRAME is currently allowed [1]
[1] https://github.com/apache/arrow/blob/master/format/Message.fbs#L45
On Tue, Jun 22, 2021 at 12:28 PM Rares Vernica <rv...@gmail.com> wrote:
> Hello,
>
> Using Arrow 3.0.0 I tried to compress a stream with LZ4 and got this error
> message:
>
> NotImplemented: Streaming compression unsupported with LZ4 raw format. Try
> using LZ4 frame format instead.
>
> Is it because LZ4 raw was not enabled when the .so was compiled or actually
> not implemented?
>
> Is LZ4 frame the recommended format for compressing stream/IPC buffers?
>
> Thank you!
> Rares
>