[jira] [Created] (FLINK-23093) Limit number of I/O pool and Future threads in Mini Cluster

["Stephan Ewen (Jira)" <ji...@apache.org>]

Stephan Ewen created FLINK-23093:
------------------------------------

             Summary: Limit number of I/O pool and Future threads in Mini Cluster
                 Key: FLINK-23093
                 URL: https://issues.apache.org/jira/browse/FLINK-23093
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Coordination
    Affects Versions: 1.13.0
            Reporter: Stephan Ewen
             Fix For: 1.14.0


When running tests on CI via the minicluster, the mini cluster typically spawns 100s of I/O threads, both in the MiniCluster I/O pool and in the TM I/O pool.

The standard rule for the maximum pool size is 4*num-cores, but the number of cores can be fairly large these days. Various Java versions also mess up core counting when running in containers (JVM container might have been given 2 cores as resource limits, but the JVM counts the system as a whole, like 64/128 cores).

This is both a nuisance for debugging, and a big waste of memory (each thread takes by default around 1MB when spawned, so the test JVM wastes 100s of MBs for nothing).

I would suggest to set a default of 8 I/O threads for the Mini Cluster. The scaling-with-cores is important for proper TM/JM deployments, but not for the Mini Cluster.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)