You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by GitBox <gi...@apache.org> on 2021/02/18 13:04:22 UTC
[GitHub] [parquet-mr] martin-g opened a new pull request #872: PARQUET-1975 Disable BrotliCodec for non-x86_64
martin-g opened a new pull request #872:
URL: https://github.com/apache/parquet-mr/pull/872
BrotliCodec fails to load on non-x86_64 CPU architectures because MeteoGroup/jbrotli does not provide native binaries
With this change brotli-codec is added as a dependency to parquet-hadoop only for amd64, i.e. x86_64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] gszadovszky merged pull request #872: PARQUET-1975 Disable BrotliCodec for non-x86_64
Posted by GitBox <gi...@apache.org>.
gszadovszky merged pull request #872:
URL: https://github.com/apache/parquet-mr/pull/872
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] martin-g commented on pull request #872: PARQUET-1975 Disable BrotliCodec for non-x86_64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #872:
URL: https://github.com/apache/parquet-mr/pull/872#issuecomment-782031331
brotli-codec provides binaries only for `darwin-x86-amd64` and `linux-x86-amd64` - those are actual folder names in the jar.
I've updated this PR as you requested but if someone tries to build on PPC64, s390x or RISC-V then (s)he will face the same problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] gszadovszky commented on pull request #872: PARQUET-1975 Disable BrotliCodec for non-x86_64
Posted by GitBox <gi...@apache.org>.
gszadovszky commented on pull request #872:
URL: https://github.com/apache/parquet-mr/pull/872#issuecomment-781965022
> The problem is org.apache.hadoop.io.compress.CompressionCodecFactory#CODEC_PROVIDERS (hadoop-common:2.10.1) - https://github.com/apache/hadoop/blob/branch-2.10.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java#L42.
> It uses ServiceLoader API to load all implementations of org.apache.hadoop.io.compress.CompressionCodec and brotli-codec fails due to the missing binary.
> So, having the dependency on the classpath breaks the **loading** of CompressionCodecFactory class and even code that does not use Brotli but any of the other compressions fails immediately.
I see, thanks for the explanation. Then, you may try something like `<arch>!arm64</arch>` instead of using `amd64`. I think `amd64` is too strict and we may end up not testing something without noticing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] martin-g commented on pull request #872: PARQUET-1975 Disable BrotliCodec for non-x86_64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #872:
URL: https://github.com/apache/parquet-mr/pull/872#issuecomment-782036274
Should I modify .travis.yml in this PR or you prefer this to be in separate PR ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] martin-g commented on pull request #872: PARQUET-1975 Disable BrotliCodec for non-x86_64
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #872:
URL: https://github.com/apache/parquet-mr/pull/872#issuecomment-781912160
The problem is org.apache.hadoop.io.compress.CompressionCodecFactory#CODEC_PROVIDERS (hadoop-common:2.10.1) - https://github.com/apache/hadoop/blob/branch-2.10.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java#L42.
It uses ServiceLoader API to load all implementations of org.apache.hadoop.io.compress.CompressionCodec and brotli-codec fails due to the missing binary.
So, having the dependency on the classpath breaks the **loading** of CompressionCodecFactory class and even code that does not use Brotli but any of the other compressions fails immediately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org