[jira] [Commented] (FLINK-8707) Excessive amount of files opened by flink task manager

["Alexander Gardner (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370219#comment-16370219 ] 

Alexander Gardner commented on FLINK-8707:
------------------------------------------

Hi Aljoscha

I'm adding some more info from our DEV box - could this just be an incorect interpretation of the LINUX "lsof" command or are there really that many file descriptors open?
Bear in mind that no flink jobs are running, this is merely just starting up the flink processes.

Note, history servers were not running on either boxes.

Commands used to start the flink cluster on (both) boxes:

/opt/app/xxxxx/dev/pkgs/flink/current/bin/jobmanager.sh start cluster
/opt/app/xxxxx/dev/pkgs/flink/current/bin/taskmanager.sh start

BEFORE START OF FLINK CLUSTER
Amount of FDs in total for the flink user (before) Flink Job Manager & Task Manager starts
Box 1: 
$ lsof | wc -l
2363

Box 2: 
$ lsof | wc -l
5309

AFTER START OF FLINK JOB MANAGER CLUSTER
Amount of FDs in total for the flink user (before) Flink Job Manager

Box 1
------------------
After starting JOBMANAGER
$ lsof | wc -l
14405

Box 2: After starting TASKMANAGER
$ lsof | wc -l
27072

Job Manager process only (pid = 30064)
lsof | grep 30064 | wc -l
12042

Task Manager process only (pid = 30984)
lsof | grep 30984 | wc -l
12428


BOX 2
------------------
After starting JOBMANAGER
$ lsof | wc -l
17425

Box 2: After starting TASKMANAGER
$ lsof | wc -l
32552

Job Manager process only (pid = 27217)
lsof | grep 27217 | wc -l
17208

Task Manager process only (pid = 30606)
lsof | grep 30606 | wc -l
14421


In some cases a socket can have two FDs and two NIO Selectors can access the same SocketChannel for say read and writes but just really just trying to understand why so many file descriptors.
More than one child process of the TaskManager is using sockets.

This maybe just be normal Flink behaviour but need a Flink committer / expert to just state this unless there's a known feature? If so, we need to alter our Production monitoring systems if this is normal / safe. 
If advised, we can reboot our Flink cluster if we're approaching the ulimit.

> Excessive amount of files opened by flink task manager
> ------------------------------------------------------
>
>                 Key: FLINK-8707
>                 URL: https://issues.apache.org/jira/browse/FLINK-8707
>             Project: Flink
>          Issue Type: Bug
>          Components: JobManager
>    Affects Versions: 1.3.2
>         Environment: NAME="Red Hat Enterprise Linux Server"
> VERSION="7.3 (Maipo)"
> Two boxes, each with a Job Manager & Task Manager, using Zookeeper for HA.
> flink.yaml below with some settings (removed exact box names) etc:
> env.log.dir: ...some dir...residing on the same box
> env.pid.dir: some dir...residing on the same box
> metrics.reporter.jmx.class: org.apache.flink.metrics.jmx.JMXReporter
> metrics.reporters: jmx
> state.backend: filesystem
> state.backend.fs.checkpointdir: file:///some_nfs_mount
> state.checkpoints.dir: file:///some_nfs_mount
> state.checkpoints.num-retained: 3
> high-availability.cluster-id: /tst
> high-availability.storageDir: file:///some_nfs_mount/ha
> high-availability: zookeeper
> high-availability.zookeeper.path.root: /flink
> high-availability.zookeeper.quorum: ...list of zookeeper boxes
> env.java.opts.jobmanager: ...some extra jar args
> jobmanager.archive.fs.dir: some dir...residing on the same box
> jobmanager.web.submit.enable: true
> jobmanager.web.tmpdir:  some dir...residing on the same box
> env.java.opts.taskmanager: some extra jar args
> taskmanager.tmp.dirs:  some dir...residing on the same box/var/tmp
> taskmanager.network.memory.min: 1024MB
> taskmanager.network.memory.max: 2048MB
> blob.storage.directory:  some dir...residing on the same box
>            Reporter: Alexander Gardner
>            Priority: Blocker
>             Fix For: 1.5.0
>
>
> ** NOTE ** - THE COMPONENT IS TASK MANAGER NOT JOB MANAGER 
> The job manager has less FDs than the task manager.
>  
> Hi
> A support alert indicated that there were a lot of open files for the boxes running Flink.
> There were 4 flink jobs that were dormant but had consumed a number of msgs from Kafka using the FlinkKafkaConsumer010.
> A simple general lsof:
> $ lsof | wc -l       ->  returned 153114 open file descriptors.
> Focusing on the TaskManager process (process ID = 12154):
> $ lsof | grep 12154 | wc -l-    > returned 129322 open FDs
> $ lsof -p 12154 | wc -l   -> returned 531 FDs
> There were 228 threads running for the task manager.
>  
> Drilling down a bit further, looking at a_inode and FIFO entries: 
> $ lsof -p 12154 | grep a_inode | wc -l = 100 FDs
> $ lsof -p 12154 | grep FIFO | wc -l  = 200 FDs
> $ /proc/12154/maps = 920 entries.
> Apart from lsof identifying lots of JARs and SOs being referenced there were also 244 child processes for the task manager process.
> Noticed that in each environment, a creep of file descriptors...are the above figures deemed excessive for the no of FDs in use? I know Flink uses Netty - is it using a separate Selector for reads & writes? 
> Additionally Flink uses memory mapped files? or direct bytebuffers are these skewing the numbers of FDs shown?
> Example of one child process ID 6633:
> java 12154 6633 dfdev 387u a_inode 0,9 0 5869 [eventpoll]
>  java 12154 6633 dfdev 388r FIFO 0,8 0t0 459758080 pipe
>  java 12154 6633 dfdev 389w FIFO 0,8 0t0 459758080 pipe
> Lasty, cannot identify yet the reason for the creep in FDs even if Flink is pretty dormant or has dormant jobs. Production nodes are not experiencing excessive amounts of throughput yet either.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)