You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Joe Witt (Jira)" <ji...@apache.org> on 2022/01/14 16:14:00 UTC
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476249#comment-17476249 ]
Joe Witt commented on NIFI-9572:
--------------------------------
There are no known issues involving bugs in provenance leading to too many open files right now so probably this is solvable/configuration or the underlying issue is likely present in older versions too.
First, we often have users report they are certain max open file settings are correct by them running ulimit -a but only then discover they ran it as a different user than what their nifi really launches as. Please double check this.
Second, what output we see complain of open file exhaustion is often just the victim that hit the limit. There could be a totally different underlying root cause (socket exhaustion for instance). Run 'lsof -p <nifi pid>' to get a full listing of all the file handles nifi has. Run this right after a restart/nifi initially comes up. Then run it periodically over time. If there is indeed a leak it will become pretty obvious pretty fast in terms of the likely culprit.
> Failed to index Provenance Events and (Too many Files)
> ------------------------------------------------------
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 1.15.2
> Reporter: mayki
> Priority: Major
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 50000
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 10000
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException: /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)