You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by Hao Luo <hl...@twitter.com.INVALID> on 2018/01/30 02:50:08 UTC

Does Parquet use LZOP compression?

Hi,
    I have a question about LZO compression in Parquet file. Does Parquet
use LZOP compression? If it uses LZOP, how does it differentiate between
LZO and LZOP? In the CompressionCodecName I only see LZO there. How do I
find information about which codec to use when decompress a dictionary page?

Thanks.
Hao

Re: Does Parquet use LZOP compression?

Posted by Ryan Blue <rb...@netflix.com.INVALID>.

You can use the parquet-cli or parquet-tools to inspect metadata, including
compression codecs.

Dictionary pages are compressed with the same compression codec that the
other pages use.

On Tue, Jan 30, 2018 at 10:19 AM, Hao Luo <hl...@twitter.com> wrote:

> Thanks Ryan.
>
> For the dictionary page, is it compressed using LZO? How do I find out
> what codec is used when compressing the dictionary page?
>
> Hao
>
> On Tue, Jan 30, 2018 at 8:28 AM, Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> Parquet doesn't use LZOP. The supported formats are the ones you see in
>> CompressionCodecName, which is used to identify the compression codec.
>>
>> You can see the file metadata, including compression, using the
>> parquet-cli
>> module's meta command.
>>
>> rb
>>
>> On Mon, Jan 29, 2018 at 6:50 PM, Hao Luo <hl...@twitter.com.invalid>
>> wrote:
>>
>> > Hi,
>> >     I have a question about LZO compression in Parquet file. Does
>> Parquet
>> > use LZOP compression? If it uses LZOP, how does it differentiate between
>> > LZO and LZOP? In the CompressionCodecName I only see LZO there. How do I
>> > find information about which codec to use when decompress a dictionary
>> > page?
>> >
>> > Thanks.
>> > Hao
>> >
>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
>


-- 
Ryan Blue
Software Engineer
Netflix

Re: Does Parquet use LZOP compression?

Posted by Hao Luo <hl...@twitter.com.INVALID>.

Thanks Ryan.

For the dictionary page, is it compressed using LZO? How do I find out what
codec is used when compressing the dictionary page?

Hao

On Tue, Jan 30, 2018 at 8:28 AM, Ryan Blue <rb...@netflix.com.invalid>
wrote:

> Parquet doesn't use LZOP. The supported formats are the ones you see in
> CompressionCodecName, which is used to identify the compression codec.
>
> You can see the file metadata, including compression, using the parquet-cli
> module's meta command.
>
> rb
>
> On Mon, Jan 29, 2018 at 6:50 PM, Hao Luo <hl...@twitter.com.invalid> wrote:
>
> > Hi,
> >     I have a question about LZO compression in Parquet file. Does Parquet
> > use LZOP compression? If it uses LZOP, how does it differentiate between
> > LZO and LZOP? In the CompressionCodecName I only see LZO there. How do I
> > find information about which codec to use when decompress a dictionary
> > page?
> >
> > Thanks.
> > Hao
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Does Parquet use LZOP compression?

Posted by Ryan Blue <rb...@netflix.com.INVALID>.

Parquet doesn't use LZOP. The supported formats are the ones you see in
CompressionCodecName, which is used to identify the compression codec.

You can see the file metadata, including compression, using the parquet-cli
module's meta command.

rb

On Mon, Jan 29, 2018 at 6:50 PM, Hao Luo <hl...@twitter.com.invalid> wrote:

> Hi,
>     I have a question about LZO compression in Parquet file. Does Parquet
> use LZOP compression? If it uses LZOP, how does it differentiate between
> LZO and LZOP? In the CompressionCodecName I only see LZO there. How do I
> find information about which codec to use when decompress a dictionary
> page?
>
> Thanks.
> Hao
>

-- 
Ryan Blue
Software Engineer
Netflix