You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Zilong Zhu (Jira)" <ji...@apache.org> on 2023/12/02 09:12:00 UTC
[jira] [Created] (HADOOP-19001) LD_LIBRARY_PATH is missing HADOOP_COMMON_LIB_NATIVE_DIR
Zilong Zhu created HADOOP-19001:
-----------------------------------
Summary: LD_LIBRARY_PATH is missing HADOOP_COMMON_LIB_NATIVE_DIR
Key: HADOOP-19001
URL: https://issues.apache.org/jira/browse/HADOOP-19001
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.2.4
Reporter: Zilong Zhu
When we run a spark job, we find that it cannot load the native library successfully.
We found a difference between hadoop2 and hadoop3.
hadoop2-Spark-System Properties：
|java.library.path|:/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|
hadoop3-Spark-System Properties：
|java.library.path|:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|
The key point is：
hadoop2-hadoop-config.sh：
HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH" <--267
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH <--268
hadoop3-hadoop-functions.sh:
hadoop_add_param HADOOP_OPTS java.library.path \
"-Djava.library.path=${JAVA_LIBRARY_PATH}"
export LD_LIBRARY_PATH <--1484
At the same time, the hadoop3 will clear all non-whitelisted environment variables.
I'm not sure if it was intentional. But it makes our spark job unable to find the native library on hadoop3.
Maybe we should modify hadoop-functions.sh(1484) and add LD_LIBRARY_PATH to the default configuration item yarn.nodemanager.env-whitelist.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org