[jira] [Created] (IMPALA-12975) Rework organization of Hadoop dependency on ARM builds

["Joe McDonnell (Jira)" <ji...@apache.org>]

Joe McDonnell created IMPALA-12975:
--------------------------------------

             Summary: Rework organization of Hadoop dependency on ARM builds
                 Key: IMPALA-12975
                 URL: https://issues.apache.org/jira/browse/IMPALA-12975
             Project: IMPALA
          Issue Type: Task
          Components: Infrastructure
            Reporter: Joe McDonnell


The hadoop binaries that we download from the CDP build number are built for x86_64. On x86_64, HADOOP_LIB_DIR and HADOOP_INCLUDE_DIR point to the CDP hadoop (i.e. HADOOP_HOME/lib and HADOOP_HOME/include). Various pieces (including the C++ build) use these environment variables to find the native libraries.

On ARM, we leave those environment variables pointed to that same location. We fix things up by downloading a separate hadoop-client built for ARM, then copying the contents into the usual location in the CDP hadoop directory, overwriting the x86_64 contents. The code to overwrite the libraries runs on each invocation of buildall.sh

On ARM, we could change this to point HADOOP_LIB_DIR to the downloaded hadoop-client (which is built for ARM). With a bit of work on the hadoop-client, we could get it to also have the header files and also point HADOOP_INCLUDE_DIR to it. This avoids the need to copy files during buildall.sh. Any build that wants to pass in a custom hadoop can then use HADOOP_LIB_DIR_OVERRIDE and HADOOP_INCLUDE_DIR_OVERRIDE.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org