You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Jonathan Hurley (JIRA)" <ji...@apache.org> on 2017/11/14 21:18:00 UTC
[jira] [Created] (AMBARI-22444) Add Native Libraries To Tez Tarball
Jonathan Hurley created AMBARI-22444:
----------------------------------------
Summary: Add Native Libraries To Tez Tarball
Key: AMBARI-22444
URL: https://issues.apache.org/jira/browse/AMBARI-22444
Project: Ambari
Issue Type: Bug
Affects Versions: 2.6.1
Reporter: Jonathan Hurley
Assignee: Jonathan Hurley
Priority: Blocker
Fix For: 2.6.1
As part of the investigation for getting patch upgrades working in Ambari 2.6.1, it was determined that the Tez tarball will need to have the hadoop native libraries added to it so that they can be detected from the tarball.
STR:
- Install ZK, MapR, Tez, Yarn, Hive
- Enable a non-LZO codec, like Snappy
- Patch Hive to a new version
- Change the following properties in {{tez-site}}:
-- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
-- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
When Hive commands run, they will attempt to load the native snappy libraries from the Tez tarball and will fail with:
{code}
Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217)
{code}
In order to fix this, the Tez tarball should include the native hadoop libraries as well:
{code}
├── tez
│ ├── lib
│ │ ├── native
│ │ │ ├── libhadoop.a
│ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
│ │ │ ├── libhadoop.so.1.0.0
│ │ │ ├── libhadooppipes.a
│ │ │ ├── libhadooputils.a
│ │ │ ├── libhdfs.a
│ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
│ │ │ ├── libhdfs.so.0.0.0
│ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
│ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
│ │ │ └── libsnappy.so.1.1.4
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)