You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Greene (US), Geoffrey N" <ge...@boeing.com> on 2023/07/03 14:30:41 UTC
Out of file descriptors? but not!
So, I came back from two weeks vacation...
My nifi (1.17.0). is misbehaving.
The logs say
2023-07-03 07:04:07,668 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
...
Caused by: java.nio.file.FileSystemException: /data/nifi-1.17.0/provenance_repository/lucene-8-index-1687383779103/_2y6.kdi: Too many open files
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
...
I'm out of file descriptors?
After determining that nifi is running as process id 19655, I see:
lsof -p 19655 | wc -l
85
85? That's really low.
/etc/sysctl.conf contains
Fs.file-max = 9815744
$ ulimit -a
address space limit (Kibytes) (-M) unlimited
core file size (blocks) (-c) 0
cpu time (seconds) (-t) unlimited
data size (Kibytes) (-d) unlimited
file size (blocks) (-f) unlimited
locks (-x) unlimited
locked address space (Kibytes) (-l) 64
message queue size (Kibytes) (-q) 800
nice (-e) 0
nofile (-n) 1048576
nproc (-u) 4096
pipe buffer size (bytes) (-p) 4096
max memory size (Kibytes) (-m) unlimited
rtprio (-r) 0
socket buffer size (bytes) (-b) 4096
sigpend (-i) 192389
stack size (Kibytes) (-s) 8192
swap size (Kibytes) (-w) not supported
threads (-T) not supported
process size (Kibytes) (-v) unlimited
I DO notice one more odd thing:
I have 0 flow files (though a number of flows running)
Despite that, I see that thre repositories aredefinitely taking up some space.
350G content_repository
712K database_repository
28K flowfile_repository
10G provenance_repository
Any thoughts? I was thinking about doing an rm -rf on provenance_repository.....or maybe decreasing
nifi.provenance.repository.max.storage.time
or increasing
nifi.provenance.repository.max.storage.size (currently set to 10 G)
in nifi.properties.
Geoffrey Greene
Re: Out of file descriptors? but not!
Posted by Mike Thomsen <mi...@gmail.com>.
I think with systemd services can have their own handler limits. Check in
the NiFi service config for that. IIRC, we had to set configuration flags
in the NiFi config specifically to force systemd to give that service
unlimited handles.
On Mon, Jul 3, 2023 at 10:31 AM Greene (US), Geoffrey N <
geoffrey.n.greene@boeing.com> wrote:
>
>
> So, I came back from two weeks vacation…
>
>
>
> My nifi (1.17.0). is misbehaving.
>
>
>
> The logs say
>
> 2023-07-03 07:04:07,668 ERROR [Index Provenance Events-1]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
>
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>
> …
>
> Caused by: java.nio.file.FileSystemException:
> /data/nifi-1.17.0/provenance_repository/lucene-8-index-1687383779103/_2y6.kdi:
> Too many open files
>
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
>
> …
>
>
>
> I’m out of file descriptors?
>
>
>
> After determining that nifi is running as process id 19655, I see:
>
>
>
> lsof -p 19655 | wc -l
>
> 85
>
>
>
> 85? That’s really low.
>
>
>
> /etc/sysctl.conf contains
>
> Fs.file-max = 9815744
>
>
>
>
>
> $ ulimit -a
>
> address space limit (Kibytes) (-M) unlimited
>
> core file size (blocks) (-c) 0
>
> cpu time (seconds) (-t) unlimited
>
> data size (Kibytes) (-d) unlimited
>
> file size (blocks) (-f) unlimited
>
> locks (-x) unlimited
>
> locked address space (Kibytes) (-l) 64
>
> message queue size (Kibytes) (-q) 800
>
> nice (-e) 0
>
> nofile (-n) 1048576
>
> nproc (-u) 4096
>
> pipe buffer size (bytes) (-p) 4096
>
> max memory size (Kibytes) (-m) unlimited
>
> rtprio (-r) 0
>
> socket buffer size (bytes) (-b) 4096
>
> sigpend (-i) 192389
>
> stack size (Kibytes) (-s) 8192
>
> swap size (Kibytes) (-w) not supported
>
> threads (-T) not supported
>
> process size (Kibytes) (-v) unlimited
>
>
>
> I DO notice one more odd thing:
>
>
>
> I have 0 flow files (though a number of flows running)
>
>
>
> Despite that, I see that thre repositories aredefinitely taking up some
> space.
>
>
>
> 350G content_repository
>
> 712K database_repository
>
> 28K flowfile_repository
>
> 10G provenance_repository
>
>
>
>
>
>
>
> Any thoughts? I was thinking about doing an rm –rf on
> provenance_repository…..or maybe decreasing
>
> nifi.provenance.repository.max.storage.time
>
> or increasing
>
> nifi.provenance.repository.max.storage.size (currently set to 10 G)
>
> in nifi.properties.
>
>
>
>
>
>
>
> Geoffrey Greene
>
>
>