You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Sebastian Mattheis (Jira)" <ji...@apache.org> on 2022/07/01 06:41:00 UTC
[jira] [Commented] (FLINK-24932) Frocksdb cannot run on Apple M1
[ https://issues.apache.org/jira/browse/FLINK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561314#comment-17561314 ]
Sebastian Mattheis commented on FLINK-24932:
--------------------------------------------
[~yunta] I have a cherry-pick of [https://github.com/facebook/rocksdb/commit/bcd86d9074c7a3c43f7623f30f67bcbf0c4a905d] on top of [https://github.com/ververica/frocksdb/tree/FRocksDB-6.20.3] for which I'm running now the tests. The cherry-pick of the Java code was not an issue but resolving conflicts in the Makefile is awkward and I'm not sure if that will work.
Other than that: [~martijnvisser] , I have yesterday built finally the last remaining binary for [https://github.com/smattheis/frocksdb/tree/FRocksDB-6.29.5] (now includes win DLL, ppc64, linux x86/arm64, Mac x86/arm64 binaries) I have collected them all in an S3 bucket and would be ready to release the new FrocksDB version and simply bump the version in Flink. (The builds and their different setups were kind of crazy to organize and setup.) However, as [~yunta] suggested I'm now trying build FRocksDB-6.20.3 with only the Java M1 fix hoping to get some better performance. If that works I can reuse my setup for building the respective binaries.
This is just a status update and I will keep you tuned ...
> Frocksdb cannot run on Apple M1
> -------------------------------
>
> Key: FLINK-24932
> URL: https://issues.apache.org/jira/browse/FLINK-24932
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Yun Tang
> Assignee: Sebastian Mattheis
> Priority: Minor
>
> After we bump up RocksDB version to 6.20.3, we support to run RocksDB on linux arm cluster. However, according to the feedback from Robert, Apple M1 machines cannot run FRocksDB yet:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-runtime-1.14.0.jar:1.14.0]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from any of the 1 provided restore options.
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.io.IOException: Could not load the native RocksDB library
> at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.lang.UnsatisfiedLinkError: /private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib: dlopen(/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib, 0x0001): tried: '/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/lib/librocksdbjni-osx.jnilib' (no such file)
> at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_312]
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950) ~[?:1.8.0_312]
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832) ~[?:1.8.0_312]
> at java.lang.Runtime.load0(Runtime.java:811) ~[?:1.8.0_312]
> at java.lang.System.load(System.java:1088) ~[?:1.8.0_312]
> at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:79) ~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
> at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:57) ~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
> at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:856) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> {code}
> This issue is tracked by RocksDB community: [rocksdb/issues/7720|https://github.com/facebook/rocksdb/issues/7720]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)