[GitHub] [arrow] yew011 opened pull request #2479: compression_lz4.cc: Detect and handle hadoop LZ4 prefix.

["yew011 (GitHub)" <gi...@apache.org>]

For hadoop LZ4 compression, the compressed data is prefixed with 4-byte
original data length (big-endian) and then 4-byte compressed data length
(big-endian).  For details, please refer to JIRA ticket:

    https://issues.apache.org/jira/browse/HADOOP-12990

This causes LZ4 compressed chunk (e.g. parquet column data page) by code
using hadoop LZ4 codec (e.g. apache/parquet-mr project) to be not parse-able
in arrow.

This commit proposes a fix which detect the first 8 bytes matching the
hadoop format and try to decompress from after the prefix.  I already
confirmed the commit work in JIRA ticket:

    https://issues.apache.org/jira/browse/PARQUET-1241

Signed-off-by: Alex Wang <al...@awakenetworks.com>

[ Full content available at: https://github.com/apache/arrow/pull/2479 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [arrow] wesm commented on issue #2479: compression_lz4.cc: Detect and handle hadoop LZ4 prefix.

["wesm (GitHub)" <gi...@apache.org>]

What I'm saying is that the Lz4Codec here is not intended for exclusive use in the Hadoop ecosystem. So we would define a codec that is suitable for use with parquet-cpp and other applications using the Hadoop conventions for lz4 compression

[ Full content available at: https://github.com/apache/arrow/pull/2479 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [arrow] yew011 commented on issue #2479: compression_lz4.cc: Detect and handle hadoop LZ4 prefix.

["yew011 (GitHub)" <gi...@apache.org>]

@wesm sure, closing this PR, and will work on adding an open source (lz4/lz4-java) lz4 implementation to parquet-mr first.  so that the generated parquet file with lz4 compression can be processed by arrow.

[ Full content available at: https://github.com/apache/arrow/pull/2479 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [arrow] yew011 commented on issue #2479: compression_lz4.cc: Detect and handle hadoop LZ4 prefix.

["yew011 (GitHub)" <gi...@apache.org>]

This is a tentative proposal, I'd also like to fix `parquet-mr`,

[ Full content available at: https://github.com/apache/arrow/pull/2479 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [arrow] wesm commented on issue #2479: compression_lz4.cc: Detect and handle hadoop LZ4 prefix.

["wesm (GitHub)" <gi...@apache.org>]

@yew011 I think we need to add a new Codec to Apache Arrow and leave this Lz4Codec independent of any Hadoop-specific details. You could call it HADOOP_LZ4 or something like that

[ Full content available at: https://github.com/apache/arrow/pull/2479 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [arrow] yew011 closed pull request #2479: compression_lz4.cc: Detect and handle hadoop LZ4 prefix.

["yew011 (GitHub)" <gi...@apache.org>]

[ pull request closed by yew011 ]

[ Full content available at: https://github.com/apache/arrow/pull/2479 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [arrow] yew011 commented on issue #2479: compression_lz4.cc: Detect and handle hadoop LZ4 prefix.

["yew011 (GitHub)" <gi...@apache.org>]

@wesm the tricky part is that parquet-mr uses `hadoop_lz4` but marks the compression metadata as `lz4`.  So arrow cannot detect the codec is actually `hadoop_lz4`.

we could the following instead:
1. change parquet-mr lz4 to use 'lz4/lz4-java' implementation, just like parquet-mr uses `org.xerial.snappy` instead of `hadoop_snappy`.

or/and

2. try to add lz4-frame codec to both parquet-mr and arrow

[ Full content available at: https://github.com/apache/arrow/pull/2479 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org