You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Polepoint (via GitHub)" <gi...@apache.org> on 2024/08/19 04:06:20 UTC
[I] [C++][Parquet] lz4-hadoop not support block compression [arrow]
Polepoint opened a new issue, #43745:
URL: https://github.com/apache/arrow/issues/43745
### Describe the bug, including details regarding any error messages, version, and platform.
According to the
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/Lz4Codec.cc
and
https://github.com/apache/hadoop/blob/release-3.4.1-RC1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java#L92
the lz4-hadoop should be implemented with block stream, which means that the input maybe split into blocks, each block will be compressed with lz4.
The outputs will be like that
`- 4 byte big endian uncompressed_size of all blocks`
`- 4-byte big endian compressed_size of the flowing block`
`< lz4 compressed block >`
`- 4-byte big endian compressed_size of the flowing block`
`< lz4 compressed block >`
`- 4-byte big endian compressed_size of the flowing block`
`< lz4 compressed block >`
`... repeated until uncompressed_size from outer block is consumed ...`
The implement of lz4-hadoop in arrow seems only accept one block, as it will return `kNotHadoop` immediately if the `maybe_decompressed_size` of the first block is not equal to `expected_ decompressed_size`(acturally, it is the size of all blocks's decompressed_size).
https://github.com/apache/arrow/blob/release-17.0.0-rc2/cpp/src/arrow/util/compression_lz4.cc#L509
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] lz4-hadoop doesn't support block compression [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #43745:
URL: https://github.com/apache/arrow/issues/43745#issuecomment-2297320758
We see here the problem with the "lz4-hadoop" codec: its spec only exists as Java source code. That's it's superseded by [LZ4_RAW](https://github.com/apache/parquet-format/blob/master/Compression.md#lz4_raw).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] lz4-hadoop doesn't support block compression [arrow]
Posted by "nanoric (via GitHub)" <gi...@apache.org>.
nanoric commented on issue #43745:
URL: https://github.com/apache/arrow/issues/43745#issuecomment-2460792489
But which format refers to LZ4_RAW?
It sesms there is only LZ4, LZ4_FRAME, LZ4_HADOOP:
https://github.com/apache/arrow/blob/9b27f42e02d9c4208698a324357cafaaa3e308ce/cpp/src/arrow/util/type_fwd.h#L46
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] lz4-hadoop doesn't support block compression [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #43745:
URL: https://github.com/apache/arrow/issues/43745#issuecomment-2461698259
Ah, sorry. LZ4_RAW is simply the same as LZ4 here.
Le 6 novembre 2024 22:12:54 GMT+01:00, nanoric ***@***.***> a écrit :
>
>But which format refers to LZ4_RAW?
>It sesms there is only LZ4, LZ4_FRAME, LZ4_HADOOP:
>https://github.com/apache/arrow/blob/9b27f42e02d9c4208698a324357cafaaa3e308ce/cpp/src/arrow/util/type_fwd.h#L46
>
>--
>Reply to this email directly or view it on GitHub:
>https://github.com/apache/arrow/issues/43745#issuecomment-2460792489
>You are receiving this because you were mentioned.
>
>Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] lz4-hadoop doesn't support block compression [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #43745:
URL: https://github.com/apache/arrow/issues/43745#issuecomment-2297315493
cc @pitrou @wgtmac does this in spec or we have similar problem before? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] lz4-hadoop doesn't support block compression [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #43745:
URL: https://github.com/apache/arrow/issues/43745#issuecomment-2298008193
This can be handled like the code below you mind. If you want this you can add it here and add a file in parquet-testing? @Polepoint ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] lz4-hadoop doesn't support block compression [arrow]
Posted by "Polepoint (via GitHub)" <gi...@apache.org>.
Polepoint commented on issue #43745:
URL: https://github.com/apache/arrow/issues/43745#issuecomment-2298657702
> This can be handled like the code below you mind. If you want this you can add it here and add a file in parquet-testing? @Polepoint ?
:ok_hand: I will try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] lz4-hadoop doesn't support block compression [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #43745:
URL: https://github.com/apache/arrow/issues/43745#issuecomment-2297999182
IIRC, LZ4_HADOOP has been deprecated in the Parquet spec. We should use LZ4_RAW in favor of LZ4_HADOOP at all times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org