回复：flink大窗口性能问题

[Hui Wang <46...@qq.com.INVALID>]

我的系统是aarch64的，在使用rocksdb作为statebackend时，出现以下错误
Caused by: java.io.IOException: Could not load the native RocksDB library
         at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:948)
         at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:489)
         at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:319)
         at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
         at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
         ... 11 more


Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-bd8659305e92a27fac27481baf57897b/librocksdbjni-linux64.so: /tmp/rocksdb-lib-bd8659305e92a27fac27481baf57897b/librocksdbjni-linux64.so: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)


         at java.lang.ClassLoader$NativeLibrary.load(Native Method)
         at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)
         at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)
         at java.lang.Runtime.load0(Runtime.java:810)
         at java.lang.System.load(System.java:1088)
         at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
         at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
         at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:923)
         ... 15 more







---原始邮件---
发件人: "Jingsong Li"<jingsonglee0@gmail.com&gt;
发送时间: 2021年7月15日(周四) 上午10:54
收件人: "user-zh"<user-zh@flink.apache.org&gt;;
主题: Re: flink大窗口性能问题


没用rocksdb吗？

On Thu, Jul 15, 2021 at 10:46 AM Michael Ran <greemqqran@163.com&gt; wrote:

&gt; 要么内存增大，或者并行增大，要么窗口改小，同时保留数据时间减少
&gt; 在 2021-07-15 10:23:25，"Hui Wang" <463329402@qq.com.INVALID&gt; 写道：
&gt; &gt;flink大窗口缓存数据量过大导致jvm频烦full gc，并且处理速度极低，最终OOM，该如何调优
&gt;


-- 
Best, Jingsong Lee