You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by "zhangzq@eastcom-sw.com" <zh...@eastcom-sw.com> on 2024/07/09 09:32:30 UTC

开窗算子open files持续增长

    从1.14升级到1.17.1后发现每隔一段时间(一两个月) 就出现too many open files异常导致任务失败
    通过 lsof -p (taskmanager pid) 发现占用的是大量的 /tmp/tm_host:port***/tmp/job_(jobid)_op_WindowOperator_*****/db 、/db/***.sst .log、/db/LOCK(deleted)

    最大20万  占用能到17万  重启相应job后会释放 但也一直在持续增长,请问有什么方法优化或者避免这情况?

    相对1.14  多了增量checkpoint, env.setStateBackend(new EmbeddedRocksDBStateBackend(true));

    job中开窗对应代码逻辑就是简单的5秒攒批后往后流
           SingleOutputStreamOperator<List<Data>> streamOperator =
                dataStream.keyBy((Data data) ->
                                data.getDataName())
                        .window(TumblingProcessingTimeWindows.of(Time.seconds(5)))
                        .apply(new WindowFunction<Data, List<Data>, String, TimeWindow>() {
                            @Override
                            public void apply(String s, TimeWindow timeWindow, Iterable<Data> input, Collector<List<Data>> out) throws Exception {
                                List<Data> list = Lists.newArrayList();
                                input.forEach(list::add);
                                out.collect(list);
                            }
                        })