You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ChenSammi (via GitHub)" <gi...@apache.org> on 2026/06/15 07:17:24 UTC
[PR] HDDS-15559. Enable cross platform hadoop native library download [ozone]
ChenSammi opened a new pull request, #10513:
URL: https://github.com/apache/ozone/pull/10513
## What changes were proposed in this pull request?
1. support cross platform hadoop native library donwload, with profile -Plinux-aarch64, -Plinux-x86_64, so build on macos, run on linux docker.
2. export native hadoop library path as DYLD_LIBRARY_PATH/LD_LIBRARY_PATH, so that ozone docker image doesn't requires include hadoop native library anymore.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15559
## How was this patch tested?
Tested locally, and with CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Enable cross platform hadoop native library download [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10513:
URL: https://github.com/apache/ozone/pull/10513#issuecomment-4765817085
> Thanks @ChenSammi for working on this. This change makes the build output depend on user's environment (OS, architecture). As discussed in #10305, this decision should be made at runtime, not build time. Build output should include all available libs, so user downloading binary tarball can use it on all platforms. (We also use the same tarball for creating Ozone Docker image.)
@adoroszlai , thanks for the review. These are the observations when I run ozone in docker in Mac M1,
1. the ozone docker is linux arm64. Both linux arm64 and x86_64 expects hadoop library with name "libhadoop.so", this symbolic file is created during build time. So it cannot support both arm64 and x86_64 at build time. Unless we move the symbolic creation at run time, but still need to detect the os/architecture at runtime.
2. The current build script, it supports auto detect os/architecture, also supports cross platform build with -P, which is needed for debug environment like mine.
3. I inspected the hadoop release package, it has normal bin package, and arm64 bin package, each only carries the native libraries for it's architecture. I feel this way is very clean. Is there any obvious benefit that mixing all platform native libraries together in one tarball?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Enable cross platform hadoop native library download [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #10513:
URL: https://github.com/apache/ozone/pull/10513#issuecomment-4977713806
My Cursor is fixing the native lib problem: https://github.com/jojochuang/ozone/commits/refs/heads/HDDS-15559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Add libhadoop to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10513:
URL: https://github.com/apache/ozone/pull/10513#discussion_r3600746620
##########
hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh:
##########
@@ -1348,6 +1349,25 @@ function ozone_os_tricks
export JAVA_HOME
fi
fi
+
+ if [ "$ARCH_TYPE" = "arm64" ]; then
+ TARGET_FILE="libhadoop_osx_aarch_64.dylib"
+ fi
+
+ pushd . > /dev/null && cd lib/native
+ # If no matching file variant was found for the current environment
+ if [ -z "$TARGET_FILE" ]; then
+ echo "Error: libhadoop doesn't support platform combination ($OS_TYPE / $ARCH_TYPE)." >&2
Review Comment:
`OS_TYPE` is deprecated, should use `OZONE_OS_TYPE`.
##########
hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh:
##########
@@ -1348,6 +1349,25 @@ function ozone_os_tricks
export JAVA_HOME
fi
fi
+
+ if [ "$ARCH_TYPE" = "arm64" ]; then
+ TARGET_FILE="libhadoop_osx_aarch_64.dylib"
+ fi
+
+ pushd . > /dev/null && cd lib/native
+ # If no matching file variant was found for the current environment
+ if [ -z "$TARGET_FILE" ]; then
+ echo "Error: libhadoop doesn't support platform combination ($OS_TYPE / $ARCH_TYPE)." >&2
+ else
+ LINK_FILE="libhadoop.dylib"
Review Comment:
By moving `LINK_FILE` definition out of `if-else`, the code block can be extracted to a function, to be reused for both `Darwin` and `Linux` cases.
##########
hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh:
##########
@@ -1348,6 +1349,25 @@ function ozone_os_tricks
export JAVA_HOME
fi
fi
+
+ if [ "$ARCH_TYPE" = "arm64" ]; then
+ TARGET_FILE="libhadoop_osx_aarch_64.dylib"
+ fi
+
+ pushd . > /dev/null && cd lib/native
+ # If no matching file variant was found for the current environment
Review Comment:
nit:
- indent is off
- `pushd && cd` is unnecessarily complex
```suggestion
fi
pushd lib/native > /dev/null
# If no matching file variant was found for the current environment
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Enable cross platform hadoop native library download [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10513:
URL: https://github.com/apache/ozone/pull/10513#issuecomment-4705540051
Clean CI https://github.com/ChenSammi/ozone/actions/runs/27526013409.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Add libhadoop to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #10513:
URL: https://github.com/apache/ozone/pull/10513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Create libhadoop symbolic link at runtime [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10513:
URL: https://github.com/apache/ozone/pull/10513#issuecomment-4991501354
Can we please keep it as draft until CI is passing in fork?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Add libhadoop to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on PR #10513:
URL: https://github.com/apache/ozone/pull/10513#issuecomment-4995023058
> My Cursor is fixing the native lib problem: https://github.com/jojochuang/ozone/commits/refs/heads/HDDS-15559/
is it a fix for m1/arm environments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Add libhadoop to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #10513:
URL: https://github.com/apache/ozone/pull/10513#discussion_r3597932195
##########
hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh:
##########
@@ -1348,6 +1349,25 @@ function ozone_os_tricks
export JAVA_HOME
fi
fi
+
+ if [ "$ARCH_TYPE" = "arm64" ]; then
+ TARGET_FILE="libhadoop_osx_aarch_64.dylib"
+ fi
+
+ pushd . > /dev/null && cd lib/native
+ # If no matching file variant was found for the current environment
+ if [ -z "$TARGET_FILE" ]; then
+ echo "Error: libhadoop doesn't support platform combination ($OS_TYPE / $ARCH_TYPE)." >&2
+ else
+ LINK_FILE="libhadoop.dylib"
+ # Check if it already exists but points to the wrong target
+ if [ -L "$LINK_FILE" ] && [ "$(readlink "$LINK_FILE")" != "$TARGET_FILE" ]; then
+ # Forcefully recreate it so it points to the correct target file
+ ln -sf "$TARGET_FILE" "$LINK_FILE" > /dev/null 2>&1
+ fi
+ export DYLD_LIBRARY_PATH=$OZONE_HOME/lib/native:$DYLD_LIBRARY_PATH
Review Comment:
would this export be effective to the current build/test execution process?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Enable cross platform hadoop native library download [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #10513:
URL: https://github.com/apache/ozone/pull/10513#issuecomment-4909456366
There's a repeated CI failure due to downloading rocksdbjni. We're updating rocksdb to 10 not sure if that's going to solve this problem. #9813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Add libhadoop to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10513:
URL: https://github.com/apache/ozone/pull/10513#discussion_r3599711110
##########
hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh:
##########
@@ -1348,6 +1349,25 @@ function ozone_os_tricks
export JAVA_HOME
fi
fi
+
+ if [ "$ARCH_TYPE" = "arm64" ]; then
+ TARGET_FILE="libhadoop_osx_aarch_64.dylib"
+ fi
+
+ pushd . > /dev/null && cd lib/native
+ # If no matching file variant was found for the current environment
+ if [ -z "$TARGET_FILE" ]; then
+ echo "Error: libhadoop doesn't support platform combination ($OS_TYPE / $ARCH_TYPE)." >&2
+ else
+ LINK_FILE="libhadoop.dylib"
+ # Check if it already exists but points to the wrong target
+ if [ -L "$LINK_FILE" ] && [ "$(readlink "$LINK_FILE")" != "$TARGET_FILE" ]; then
+ # Forcefully recreate it so it points to the correct target file
+ ln -sf "$TARGET_FILE" "$LINK_FILE" > /dev/null 2>&1
+ fi
+ export DYLD_LIBRARY_PATH=$OZONE_HOME/lib/native:$DYLD_LIBRARY_PATH
Review Comment:
This is mainly for the user of final ozone bin package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Enable cross platform hadoop native library download [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10513:
URL: https://github.com/apache/ozone/pull/10513#issuecomment-4766027236
> Is there any obvious benefit that mixing all platform native libraries together in one tarball?
Yes:
- reduced time at release (one build)
- reduced storage (no duplicated 500M+ artifacts for a difference in native library less than 1MB)
`ratis-thirdparty` (and Ozone) binaries include Netty native libraries for all platforms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Add libhadoop to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10513:
URL: https://github.com/apache/ozone/pull/10513#issuecomment-5018369174
Thanks @adoroszlai @jojochuang @yandrey321 , for the review and good suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Add libhadoop to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10513:
URL: https://github.com/apache/ozone/pull/10513#discussion_r3599711110
##########
hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh:
##########
@@ -1348,6 +1349,25 @@ function ozone_os_tricks
export JAVA_HOME
fi
fi
+
+ if [ "$ARCH_TYPE" = "arm64" ]; then
+ TARGET_FILE="libhadoop_osx_aarch_64.dylib"
+ fi
+
+ pushd . > /dev/null && cd lib/native
+ # If no matching file variant was found for the current environment
+ if [ -z "$TARGET_FILE" ]; then
+ echo "Error: libhadoop doesn't support platform combination ($OS_TYPE / $ARCH_TYPE)." >&2
+ else
+ LINK_FILE="libhadoop.dylib"
+ # Check if it already exists but points to the wrong target
+ if [ -L "$LINK_FILE" ] && [ "$(readlink "$LINK_FILE")" != "$TARGET_FILE" ]; then
+ # Forcefully recreate it so it points to the correct target file
+ ln -sf "$TARGET_FILE" "$LINK_FILE" > /dev/null 2>&1
+ fi
+ export DYLD_LIBRARY_PATH=$OZONE_HOME/lib/native:$DYLD_LIBRARY_PATH
Review Comment:
This is mainly for the user of final ozone bin package, and also for the integration test to pass in CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Add libhadoop to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10513:
URL: https://github.com/apache/ozone/pull/10513#discussion_r3599711110
##########
hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh:
##########
@@ -1348,6 +1349,25 @@ function ozone_os_tricks
export JAVA_HOME
fi
fi
+
+ if [ "$ARCH_TYPE" = "arm64" ]; then
+ TARGET_FILE="libhadoop_osx_aarch_64.dylib"
+ fi
+
+ pushd . > /dev/null && cd lib/native
+ # If no matching file variant was found for the current environment
+ if [ -z "$TARGET_FILE" ]; then
+ echo "Error: libhadoop doesn't support platform combination ($OS_TYPE / $ARCH_TYPE)." >&2
+ else
+ LINK_FILE="libhadoop.dylib"
+ # Check if it already exists but points to the wrong target
+ if [ -L "$LINK_FILE" ] && [ "$(readlink "$LINK_FILE")" != "$TARGET_FILE" ]; then
+ # Forcefully recreate it so it points to the correct target file
+ ln -sf "$TARGET_FILE" "$LINK_FILE" > /dev/null 2>&1
+ fi
+ export DYLD_LIBRARY_PATH=$OZONE_HOME/lib/native:$DYLD_LIBRARY_PATH
Review Comment:
This is mainly for the Mac user of final ozone bin package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15559. Add libhadoop to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10513:
URL: https://github.com/apache/ozone/pull/10513#issuecomment-4993228077
@adoroszlai @jojochuang , could you help to review the patch again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org