You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Greene (US), Geoffrey N" <ge...@boeing.com> on 2023/07/03 14:30:41 UTC

Out of file descriptors? but not!

So, I came back from two weeks vacation...

My nifi (1.17.0). is misbehaving.

The logs say
2023-07-03 07:04:07,668 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
...
Caused by: java.nio.file.FileSystemException: /data/nifi-1.17.0/provenance_repository/lucene-8-index-1687383779103/_2y6.kdi: Too many open files
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
...

I'm out of file descriptors?

After determining that nifi is running as process id 19655, I see:

lsof -p 19655 | wc -l
85

85?  That's really low.

/etc/sysctl.conf contains
Fs.file-max = 9815744


$ ulimit -a
address space limit (Kibytes)  (-M)  unlimited
core file size (blocks)        (-c)  0
cpu time (seconds)             (-t)  unlimited
data size (Kibytes)            (-d)  unlimited
file size (blocks)             (-f)  unlimited
locks                          (-x)  unlimited
locked address space (Kibytes) (-l)  64
message queue size (Kibytes)   (-q)  800
nice                           (-e)  0
nofile                         (-n)  1048576
nproc                          (-u)  4096
pipe buffer size (bytes)       (-p)  4096
max memory size (Kibytes)      (-m)  unlimited
rtprio                         (-r)  0
socket buffer size (bytes)     (-b)  4096
sigpend                        (-i)  192389
stack size (Kibytes)           (-s)  8192
swap size (Kibytes)            (-w)  not supported
threads                        (-T)  not supported
process size (Kibytes)         (-v)  unlimited

I DO notice one more odd thing:

I have 0 flow files (though a number of flows running)

Despite that,  I see that thre repositories aredefinitely taking up some space.

350G    content_repository
712K    database_repository
28K     flowfile_repository
10G     provenance_repository



Any thoughts? I was thinking about doing an rm -rf on provenance_repository.....or maybe decreasing
nifi.provenance.repository.max.storage.time
or increasing
nifi.provenance.repository.max.storage.size (currently set to 10 G)
in nifi.properties.



Geoffrey Greene


Re: Out of file descriptors? but not!

Posted by Mike Thomsen <mi...@gmail.com>.
I think with systemd services can have their own handler limits. Check in
the NiFi service config for that. IIRC, we had to set configuration flags
in the NiFi config specifically to force systemd to give that service
unlimited handles.

On Mon, Jul 3, 2023 at 10:31 AM Greene (US), Geoffrey N <
geoffrey.n.greene@boeing.com> wrote:

>
>
> So, I came back from two weeks vacation…
>
>
>
> My nifi (1.17.0). is misbehaving.
>
>
>
> The logs say
>
> 2023-07-03 07:04:07,668 ERROR [Index Provenance Events-1]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
>
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>
> …
>
> Caused by: java.nio.file.FileSystemException:
> /data/nifi-1.17.0/provenance_repository/lucene-8-index-1687383779103/_2y6.kdi:
> Too many open files
>
>         at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
>
> …
>
>
>
> I’m out of file descriptors?
>
>
>
> After determining that nifi is running as process id 19655, I see:
>
>
>
> lsof -p 19655 | wc -l
>
> 85
>
>
>
> 85?  That’s really low.
>
>
>
> /etc/sysctl.conf contains
>
> Fs.file-max = 9815744
>
>
>
>
>
> $ ulimit -a
>
> address space limit (Kibytes)  (-M)  unlimited
>
> core file size (blocks)        (-c)  0
>
> cpu time (seconds)             (-t)  unlimited
>
> data size (Kibytes)            (-d)  unlimited
>
> file size (blocks)             (-f)  unlimited
>
> locks                          (-x)  unlimited
>
> locked address space (Kibytes) (-l)  64
>
> message queue size (Kibytes)   (-q)  800
>
> nice                           (-e)  0
>
> nofile                         (-n)  1048576
>
> nproc                          (-u)  4096
>
> pipe buffer size (bytes)       (-p)  4096
>
> max memory size (Kibytes)      (-m)  unlimited
>
> rtprio                         (-r)  0
>
> socket buffer size (bytes)     (-b)  4096
>
> sigpend                        (-i)  192389
>
> stack size (Kibytes)           (-s)  8192
>
> swap size (Kibytes)            (-w)  not supported
>
> threads                        (-T)  not supported
>
> process size (Kibytes)         (-v)  unlimited
>
>
>
> I DO notice one more odd thing:
>
>
>
> I have 0 flow files (though a number of flows running)
>
>
>
> Despite that,  I see that thre repositories aredefinitely taking up some
> space.
>
>
>
> 350G    content_repository
>
> 712K    database_repository
>
> 28K     flowfile_repository
>
> 10G     provenance_repository
>
>
>
>
>
>
>
> Any thoughts? I was thinking about doing an rm –rf on
> provenance_repository…..or maybe decreasing
>
> nifi.provenance.repository.max.storage.time
>
> or increasing
>
> nifi.provenance.repository.max.storage.size (currently set to 10 G)
>
> in nifi.properties.
>
>
>
>
>
>
>
> Geoffrey Greene
>
>
>