You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Sebastian Mattheis (Jira)" <ji...@apache.org> on 2022/07/18 15:02:00 UTC
[jira] [Comment Edited] (FLINK-24932) Frocksdb cannot run on Apple M1
[ https://issues.apache.org/jira/browse/FLINK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17568049#comment-17568049 ]
Sebastian Mattheis edited comment on FLINK-24932 at 7/18/22 3:01 PM:
---------------------------------------------------------------------
[~yunta] Building FRocksDB on M1 silicon is not the problem. However, building the binaries for PPC64 poses a problem as Travis CI pipeline failed repeatedly due timeouts where various attempts to avoid or work around the timeout did not work: [https://app.travis-ci.com/github/smattheis/frocksdb/builds] (I have no other PPC64 system to use and QEMU emulation did not work on neither on Linux x86_64 and nor on Mac OS x86_64 or arm64).
Other than that, is anything wrong with the branch that I created [https://github.com/smattheis/frocksdb/tree/FRocksDB-6.20.3-M1-fix] ? It would have been helpful to get some review comments instead of creating a new branch.
Also I have been working on this issue on side out of courtesy, as this issue has no priority for my employer and I have actually no time to allocate for it. As you added some tests that I didn't add and you have some more urgency, I would suggest it makes more sense that you continue with your branch on this issue where doing cross-platform builds and following release process remains. [~yunta] Could you please assign the issue to yourself?
was (Author: JIRAUSER284806):
[~yunta] Building FRocksDB on M1 silicon is not the problem. However, building the binaries for PPC64 poses a problem as Travis CI pipeline failed repeatedly due timeouts where various attempts to avoid or work around the timeout did not work: [https://app.travis-ci.com/github/smattheis/frocksdb/builds] (I have no other PPC64 system to use and QEMU emulation did not work on neither on Linux x86_64 and nor on Mac OS x86_64 or arm64).
Other than that, is anything wrong with the branch that I created [https://github.com/smattheis/frocksdb/tree/FRocksDB-6.20.3-M1-fix] ? It would have been helpful to get some review comments instead of creating a new branch.
Also I have been working on this issue on side out of courtesy, as this issue has no priority for my employer and I have actually no time to allocate for it. As you added some tests that I didn't add and have some more urgency, I would suggest it makes more sense that you continue with your branch on this issue where doing cross-platform builds and following release process remains. [~yunta] Could you please assign the issue to yourself?
> Frocksdb cannot run on Apple M1
> -------------------------------
>
> Key: FLINK-24932
> URL: https://issues.apache.org/jira/browse/FLINK-24932
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Yun Tang
> Assignee: Sebastian Mattheis
> Priority: Minor
>
> After we bump up RocksDB version to 6.20.3, we support to run RocksDB on linux arm cluster. However, according to the feedback from Robert, Apple M1 machines cannot run FRocksDB yet:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-runtime-1.14.0.jar:1.14.0]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from any of the 1 provided restore options.
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.io.IOException: Could not load the native RocksDB library
> at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.lang.UnsatisfiedLinkError: /private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib: dlopen(/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib, 0x0001): tried: '/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/lib/librocksdbjni-osx.jnilib' (no such file)
> at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_312]
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950) ~[?:1.8.0_312]
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832) ~[?:1.8.0_312]
> at java.lang.Runtime.load0(Runtime.java:811) ~[?:1.8.0_312]
> at java.lang.System.load(System.java:1088) ~[?:1.8.0_312]
> at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:79) ~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
> at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:57) ~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
> at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:856) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> {code}
> This issue is tracked by RocksDB community: [rocksdb/issues/7720|https://github.com/facebook/rocksdb/issues/7720]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)