You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "mayki (Jira)" <ji...@apache.org> on 2022/01/17 12:39:00 UTC
[jira] [Comment Edited] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477179#comment-17477179 ]
mayki edited comment on NIFI-9572 at 1/17/22, 12:38 PM:
--------------------------------------------------------
I also remove all files in the repository
{code:java}
rm -fr content_repository/*
rm -fr database_repository/*
rm -fr flowfile_repository/*
rm -fr provenance_repository/* {code}
Now it failed to start
{code:java}
2022-01-17 13:36:01,040 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.makePipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at java.nio.channels.Selector.open(Selector.java:227)
at org.eclipse.jetty.io.SelectorManager.newSelector(SelectorManager.java:383)
at org.eclipse.jetty.io.ManagedSelector.doStart(ManagedSelector.java:115)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:262)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1142)
at org.apache.nifi.NiFi.<init>(NiFi.java:170)
at org.apache.nifi.NiFi.<init>(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
2022-01-17 13:36:01,043 INFO [Thread-2] org.apache.nifi.NiFi Application Server shutdown started
2022-01-17 13:36:01,047 INFO [Thread-2] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@4387d2f4{SSL, (ssl, http/1.1)}
2022-01-17 13:36:01,047 INFO [Thread-2] org.eclipse.jetty.server.session node0 Stopped scavenging
{code}
{code:java}
# lsof -p 775865 | wc -l
8052
# ls -rtl /proc/775865/fd | wc -l
4097
{code}
was (Author: mayki):
I also remove all files in the repository
{code:java}
rm -fr content_repository/*
rm -fr database_repository/*
rm -fr flowfile_repository/*
rm -fr provenance_repository/* {code}
Now it failed to start
{code:java}
2022-01-17 13:36:01,040 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.makePipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at java.nio.channels.Selector.open(Selector.java:227)
at org.eclipse.jetty.io.SelectorManager.newSelector(SelectorManager.java:383)
at org.eclipse.jetty.io.ManagedSelector.doStart(ManagedSelector.java:115)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:262)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1142)
at org.apache.nifi.NiFi.<init>(NiFi.java:170)
at org.apache.nifi.NiFi.<init>(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
2022-01-17 13:36:01,043 INFO [Thread-2] org.apache.nifi.NiFi Application Server shutdown started
2022-01-17 13:36:01,047 INFO [Thread-2] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@4387d2f4{SSL, (ssl, http/1.1)}{s3575tos.mc2.renault.fr:9091}
2022-01-17 13:36:01,047 INFO [Thread-2] org.eclipse.jetty.server.session node0 Stopped scavenging
{code}
{code:java}
# lsof -p 775865 | wc -l
8052
# ls -rtl /proc/775865/fd | wc -l
4097
{code}
> Failed to index Provenance Events and (Too many Files)
> ------------------------------------------------------
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 1.15.2
> Reporter: mayki
> Priority: Major
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 50000
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 10000
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException: /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)