[jira] [Commented] (BEAM-11231) Better support for Hadoop native libs and compression codecs in Java Dataflow batch worker

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299742#comment-17299742 ] 

Beam JIRA Bot commented on BEAM-11231:
--------------------------------------

This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3.

Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean.


> Better support for Hadoop native libs and compression codecs in Java Dataflow batch worker
> ------------------------------------------------------------------------------------------
>
>                 Key: BEAM-11231
>                 URL: https://issues.apache.org/jira/browse/BEAM-11231
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-gcp, io-java-hadoop-file-system, io-java-hadoop-format, io-java-hbase, runner-dataflow
>    Affects Versions: 2.24.0
>            Reporter: Cheng Li
>            Priority: P2
>              Labels: GCP, stale-P2
>
> Current(as of Beam SDK 2.24.0) Java Dataflow workers does not ship with [Hadoop native library|https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-common/NativeLibraries.html] and popular native compression codecs(e.g libsnappy) .
> If one try to read files using IO classes that assume native library support(e.g, org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat) on GCP Dataflow, the following error will be throw if the file is compressed by snappy:
>  
> {code}
> java.io.IOException: Failed to start reading from source: org.apache.beam.sdk.io.hadoop.format.HadoopFormatIO$HadoopInputFor
>  matBoundedSource@311fa603
> ...
> Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
>   
> {code}
> There is very little one can do from within pipeline job code unless rewriting the IO class, so it might make sense to include these native libraries for Hadoop IOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)