You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Jonathan Swenson (Jira)" <ji...@apache.org> on 2022/07/05 17:40:00 UTC
[jira] [Commented] (ARROW-16608) [Gandiva][Java] Unsatisfied Link Error on M1 Mac when using mavencentral artifacts
[ https://issues.apache.org/jira/browse/ARROW-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17562776#comment-17562776 ]
Jonathan Swenson commented on ARROW-16608:
------------------------------------------
{{I was toying with adding the {color:#505f79}CMAKE_OSX_ARCHITECTURES{color} flag to see if I could get a simple universal binary built. }}
{code:java}
cmake \
-DCMAKE_OSX_ARCHITECTURES='arm64;x86_64' \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=../java-dist \
../java/c {code}
building (following the rest of the build commands in [this doc|https://arrow.apache.org/docs/dev/developers/java/building.html#building-jni-libraries-on-macos]) succeeds with the following warnings:
{code:java}
/Users/me/src/arrow/java/c/src/main/cpp/jni_wrapper.cc:201:3: warning: 'auto' type specifier is a C++11 extension [-Wc++11-extensions]
auto arr = reinterpret_cast<jbyteArray>(error_data);
^ {code}
{code:java}
[100%] Linking CXX shared library libarrow_cdata_jni.dylib
ld: warning: ignoring file /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/server/libjvm.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
[100%] Built target arrow_cdata_jni
Install the project...
-- Install configuration: "Release"
-- Installing: /Users/me/src/arrow/java-dist/lib/libarrow_cdata_jni.dylib {code}
The build produces the lib_arrow-cdata_jni.dylib which appears to have both the x64 and arm64 binaries within:
{code:java}
❯ file /Users/me/src/arrow/java-dist/lib/libarrow_cdata_jni.dylib
/Users/me/src/arrow/java-dist/lib/libarrow_cdata_jni.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64Mach-O 64-bit dynamically linked shared library x86_64] [arm64:Mach-O 64-bit dynamically linked shared library arm64Mach-O 64-bit dynamically linked shared library arm64]
/Users/me/src/arrow/java-dist/lib/libarrow_cdata_jni.dylib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64
/Users/me/src/arrow/java-dist/lib/libarrow_cdata_jni.dylib (for architecture arm64): Mach-O 64-bit dynamically linked shared library arm64 {code}
However when I try to do the same for the other JNI libraries (gandiva, orc, etc) I get an error running the initial cmake:
{code:java}
❯ cmake \
-DCMAKE_OSX_ARCHITECTURES='arm64;x86_64' \
-DARROW_BOOST_USE_SHARED=OFF \
-DARROW_BROTLI_USE_SHARED=OFF \
-DARROW_BZ2_USE_SHARED=OFF \
-DARROW_GFLAGS_USE_SHARED=OFF \
-DARROW_GRPC_USE_SHARED=OFF \
-DARROW_LZ4_USE_SHARED=OFF \
-DARROW_OPENSSL_USE_SHARED=OFF \
-DARROW_PROTOBUF_USE_SHARED=OFF \
-DARROW_SNAPPY_USE_SHARED=OFF \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
-DARROW_JNI=ON \
-DARROW_PARQUET=ON \
-DARROW_FILESYSTEM=ON \
-DARROW_DATASET=ON \
-DARROW_GANDIVA_JAVA=ON \
-DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \
-DARROW_GANDIVA=ON \
-DARROW_ORC=ON \
-DARROW_PLASMA_JAVA_CLIENT=ON \
-DARROW_PLASMA=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=../java-dist \
-DCMAKE_UNITY_BUILD=ON \
-Dre2_SOURCE=BUNDLED \
-DBoost_SOURCE=BUNDLED \
-Dutf8proc_SOURCE=BUNDLED \
-DSnappy_SOURCE=BUNDLED \
-DORC_SOURCE=BUNDLED \
-DZLIB_SOURCE=BUNDLED \
../cpp
-- Building using CMake version: 3.23.2
-- The C compiler identification is AppleClang 13.1.6.13160021
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Arrow version: 9.0.0 (full: '9.0.0-SNAPSHOT')
-- Arrow SO version: 900 (full: 900.0.0)
-- clang-tidy 12 not found
-- clang-format 12 not found
-- Could NOT find ClangTools (missing: CLANG_FORMAT_BIN CLANG_TIDY_BIN)
-- infer not found
-- Found Python3: /opt/homebrew/bin/python3.9 (found version "3.9.13") found components: Interpreter
-- Using ccache: /opt/homebrew/bin/ccache
-- Found cpplint executable at /Users/me/src/arrow/cpp/build-support/cpplint.py
-- System processor: arm64
-- Performing Test CXX_SUPPORTS_ARMV8_ARCH
-- Performing Test CXX_SUPPORTS_ARMV8_ARCH - Failed
-- Arrow build warning level: PRODUCTION
CMake Error at cmake_modules/SetupCxxFlags.cmake:477 (message):
Unsupported arch flag: -march=armv8-a.
Call Stack (most recent call first):
CMakeLists.txt:494 (include){code}
I get a little further if I run a cmake without that flag first, but then the actual build step fails:
{code:java}
42%] Built target arrow_dependencies
[ 43%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/Unity/unity_24_cxx.cxx.o
error: unknown target CPU 'armv8-a'
note: valid target CPU values are: nocona, core2, penryn, bonnell, atom, silvermont, slm, goldmont, goldmont-plus, tremont, nehalem, corei7, westmere, sandybridge, corei7-avx, ivybridge, core-avx-i, haswell, core-avx2, broadwell, skylake, skylake-avx512, skx, cascadelake, cooperlake, cannonlake, icelake-client, rocketlake, icelake-server, tigerlake, sapphirerapids, alderlake, knl, knm, k8, athlon64, athlon-fx, opteron, k8-sse3, athlon64-sse3, opteron-sse3, amdfam10, barcelona, btver1, btver2, bdver1, bdver2, bdver3, bdver4, znver1, znver2, znver3, x86-64, x86-64-v2, x86-64-v3, x86-64-v4
make[2]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/Unity/unity_24_cxx.cxx.o] Error 1
make[1]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/all] Error 2
make: *** [all] Error 2 {code}
Perhaps related to https://issues.apache.org/jira/browse/ARROW-9744
I don't have a great handle on what's actually going to work here, but I thought maybe I'd try a few options.
> [Gandiva][Java] Unsatisfied Link Error on M1 Mac when using mavencentral artifacts
> ----------------------------------------------------------------------------------
>
> Key: ARROW-16608
> URL: https://issues.apache.org/jira/browse/ARROW-16608
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++ - Gandiva, Java
> Affects Versions: 8.0.0
> Reporter: Jonathan Swenson
> Assignee: Anthony Louis Gotlib Ferreira
> Priority: Major
>
> Potentially a blocker for Arrow Integration into Calcite: CALCITE-2040, however it may be possible to move forward without M1 Mac support.
> potentially somewhat related to ARROW-11135
> Getting an instance of the JNILoader throw a Unsatisfied Link Error when it tries to load the libgandiva_jni.dylib that it has extracted from the jar into a temporary directory.
> Simplified error:
> {code:java}
> Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp_dir/libgandiva_jni.dylib_uuid: dlopen(/tmp_dir/libgandiva_jni.dylib_uuid, 0x0001): tried: '/tmp_dir/libgandiva_jni.dylib_uuid' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')){code}
>
> Full error and stack trace:
> {code:java}
> Exception in thread "main" java.lang.UnsatisfiedLinkError: /private/var/folders/fj/63_6n5dx10n4b5x7jtdj6tvh0000gn/T/libgandiva_jni.dylib526a47e1-7306-440f-8bbf-378877abe5fe: dlopen(/private/var/folders/fj/63_6n5dx10n4b5x7jtdj6tvh0000gn/T/libgandiva_jni.dylib526a47e1-7306-440f-8bbf-378877abe5fe, 0x0001): tried: '/private/var/folders/fj/63_6n5dx10n4b5x7jtdj6tvh0000gn/T/libgandiva_jni.dylib526a47e1-7306-440f-8bbf-378877abe5fe' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
> at java.lang.Runtime.load0(Runtime.java:811)
> at java.lang.System.load(System.java:1088)
> at org.apache.arrow.gandiva.evaluator.JniLoader.loadGandivaLibraryFromJar(JniLoader.java:74)
> at org.apache.arrow.gandiva.evaluator.JniLoader.setupInstance(JniLoader.java:63)
> at org.apache.arrow.gandiva.evaluator.JniLoader.getInstance(JniLoader.java:53)
> at org.apache.arrow.gandiva.evaluator.JniLoader.getDefaultConfiguration(JniLoader.java:144)
> at org.apache.arrow.gandiva.evaluator.Filter.make(Filter.java:67)
> at io.acme.Main.main(Main.java:26) {code}
>
> This example loads three libraries from mavencentral using gradle:
> {code:java}
> repositories {
> mavenCentral()
> }
> dependencies {
> implementation("org.apache.arrow:arrow-memory-netty:8.0.0")
> implementation("org.apache.arrow:arrow-vector:8.0.0")
> implementation("org.apache.arrow.gandiva:arrow-gandiva:8.0.0")
> } {code}
> Example code:
> {code:java}
> public class Main {
> public static void main(String[] args) throws GandivaException {
> Field field = new Field("int_field", FieldType.nullable(new ArrowType.Int(32, true)), null);
> Schema schema = makeSchema(field);
> Condition condition = makeCondition(field);
> Filter.make(schema, condition);
> }
> private static Schema makeSchema(Field field) {
> List<Field> fieldList = new ArrayList<>();
> fieldList.add(field);
> return new Schema(fieldList, null);
> }
> private static Condition makeCondition(Field f) {
> List<TreeNode> treeNodes = new ArrayList<>(2);
> treeNodes.add(TreeBuilder.makeField(f));
> treeNodes.add(TreeBuilder.makeLiteral(4));
> TreeNode comparison = TreeBuilder.makeFunction("less_than", treeNodes, new ArrowType.Bool());
> return TreeBuilder.makeCondition(comparison);
> }
> } {code}
> While I haven't tested this exact example, a similar example executes without issue on an intel x86 mac.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)