You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by "zhangzq@eastcom-sw.com" <zh...@eastcom-sw.com> on 2024/07/09 09:32:30 UTC
开窗算子open files持续增长
从1.14升级到1.17.1后发现每隔一段时间(一两个月) 就出现too many open files异常导致任务失败
通过 lsof -p (taskmanager pid) 发现占用的是大量的 /tmp/tm_host:port***/tmp/job_(jobid)_op_WindowOperator_*****/db 、/db/***.sst .log、/db/LOCK(deleted)
最大20万 占用能到17万 重启相应job后会释放 但也一直在持续增长,请问有什么方法优化或者避免这情况?
相对1.14 多了增量checkpoint, env.setStateBackend(new EmbeddedRocksDBStateBackend(true));
job中开窗对应代码逻辑就是简单的5秒攒批后往后流
SingleOutputStreamOperator<List<Data>> streamOperator =
dataStream.keyBy((Data data) ->
data.getDataName())
.window(TumblingProcessingTimeWindows.of(Time.seconds(5)))
.apply(new WindowFunction<Data, List<Data>, String, TimeWindow>() {
@Override
public void apply(String s, TimeWindow timeWindow, Iterable<Data> input, Collector<List<Data>> out) throws Exception {
List<Data> list = Lists.newArrayList();
input.forEach(list::add);
out.collect(list);
}
})