You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "advancedxy (via GitHub)" <gi...@apache.org> on 2024/02/26 03:38:10 UTC
[PR] fix: Another attempt to fix libcrypto.dylib loading issue [arrow-datafusion-comet]
advancedxy opened a new pull request, #112:
URL: https://github.com/apache/arrow-datafusion-comet/pull/112
## Which issue does this PR close?
Closes #76
## Rationale for this change
The previous fix(#55) is not correct as it doesn't propagate `DYLD_LIBRARY_PATH` to the test phase: the `export` statement is executed in a bash shell, which is lost in the new shell command in the test phase.
Q: If the library path is not correctly set, how does most of the test instances are passed?
A: It turns out that Github's mac runners have inconsistent state of SIP enable status. See https://github.com/actions/runner-images/issues/9091 for related reports, and I also observed related output in my personal debug branch. When SIP is disabled, the X86 mac runners will try to find `libcrypto.dylib` in `/usr/lib/local` on macos-latest(a.k.a: macos-12), which are already linked by homebrew's openssl lib. Below is a screenshot that listing `/usr/local/lib`:
<img width="1067" alt="image" src="https://github.com/apache/arrow-datafusion-comet/assets/807537/ff84f433-a7b4-4054-ba95-eb2015ba2ba0">
So all the tests passed are ran on Mac with SIP disabled.
--------
In theory, even if SIP is turned on, the java binary from zulu distribution should also try to find libraries in `/usr/local/lib` if it's unrestricted[1]. I am not sure and didn't find a clue why it doesn't and fails the test.
-------
Q: How does upgrade to macos-13 fix this problem?
A: macos-13's SIP is turned off, See: https://github.com/actions/runner-images/issues/8162. So it's possible to manipulate the `DYLD_LIBARARY_PATH` for our Java tests for now.
## What changes are included in this PR?
1. upgrade X86 mac runners to macos-13
2. correctly setup `DYLD_LIBRARY_PATH`
## How are these changes tested?
Manual verification
Notes:
1: you can look up related documentation with `man dlopen` on your Mac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Another attempt to fix libcrypto.dylib loading issue [arrow-datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #112:
URL: https://github.com/apache/arrow-datafusion-comet/pull/112#issuecomment-1964661563
Merged, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Another attempt to fix libcrypto.dylib loading issue [arrow-datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #112:
URL: https://github.com/apache/arrow-datafusion-comet/pull/112#discussion_r1502998914
##########
.github/actions/setup-macos-builder/action.yaml:
##########
@@ -49,12 +49,13 @@ runs:
unzip $PROTO_ZIP
echo "$HOME/d/protoc/bin" >> $GITHUB_PATH
export PATH=$PATH:$HOME/d/protoc/bin
- # install openssl and setup DYLD_LIBRARY_PATH to work with libcrypto.dylib loading issues with x86_64 mac runners
- # see PR https://github.com/apache/arrow-datafusion-comet/pull/55 for more details
+ # install openssl and setup DYLD_LIBRARY_PATH
brew install openssl
- OPENSSL_LIB_PATH=$(dirname `brew list openssl | grep 'lib/libcrypto.dylib'`)
+ OPENSSL_LIB_PATH=`brew --prefix openssl`/lib
echo "openssl lib path is: ${OPENSSL_LIB_PATH}"
- export DYLD_LIBRARY_PATH=$OPENSSL_LIB_PATH:$DYLD_LIBRARY_PATH
+ echo "DYLD_LIBRARY_PATH=$OPENSSL_LIB_PATH:$DYLD_LIBRARY_PATH" >> $GITHUB_ENV
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Another attempt to fix libcrypto.dylib loading issue [arrow-datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #112:
URL: https://github.com/apache/arrow-datafusion-comet/pull/112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org