You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2021/03/18 14:43:00 UTC
[jira] [Commented] (NIFI-8339) Input Threads get Interrupted and
stuck indefinitely
[ https://issues.apache.org/jira/browse/NIFI-8339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304195#comment-17304195 ]
Mark Payne commented on NIFI-8339:
----------------------------------
[~renewei] can you check your logs for the message "Unable to write to container"? I am guessing that you will see a message along the lines of:
{code:java}
2021-03-18 10:26:20,988 INFO [Timer-Driven Process Thread-10] o.a.n.c.repository.FileSystemRepository Unable to write to container default due to archive file size constraints; waiting for archive cleanup {code}
This would indicate that your content repository is filling up and is exerting backpressure on the Input Port, which prevents it from writing the data out. This can potentially result in the transfer timing out.
By default, the content repository exerts backpressure if the disk is 50% full. This can be configured in nifi.properties:
{code:java}
nifi.content.repository.archive.max.usage.percentage=50% {code}
> Input Threads get Interrupted and stuck indefinitely
> ----------------------------------------------------
>
> Key: NIFI-8339
> URL: https://issues.apache.org/jira/browse/NIFI-8339
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.13.0
> Reporter: Rene Weidlinger
> Priority: Major
> Attachments: firefox_Yf6NUeQe5X.png
>
>
> After some seconds we see this stack trace in nifi on one of our inputs:
> {noformat}
> 2021-03-18 07:33:34,703 ERROR [NiFi Web Server-18] o.a.nifi.web.api.ApplicationResource Unexpected exception occurred. portId=c4d93fb6-5e5b-1382-b39b-66fbc04660f0
> 2021-03-18 07:33:34,703 ERROR [NiFi Web Server-18] o.a.nifi.web.api.ApplicationResource Exception detail:
> org.apache.nifi.processor.exception.ProcessException: org.apache.nifi.processor.exception.ProcessException: Interrupted while waiting for site-to-site request to be serviced
> at org.apache.nifi.remote.StandardPublicPort.receiveFlowFiles(StandardPublicPort.java:588)
> at org.apache.nifi.web.api.DataTransferResource.receiveFlowFiles(DataTransferResource.java:277)
> at sun.reflect.GeneratedMethodAccessor198.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
> at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
> at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
> at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
> at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
> at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
> at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
> at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
> at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
> at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
> at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
> at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
> at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
> at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:61)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:61)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:61)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:61)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:100)
> at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
> at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
> at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
> at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487)
> at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336)
> at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.apache.nifi.web.security.headers.XContentTypeOptionsFilter.doFilter(XContentTypeOptionsFilter.java:48)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:766)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
> at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
> at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.nifi.processor.exception.ProcessException: Interrupted while waiting for site-to-site request to be serviced
> at org.apache.nifi.remote.StandardPublicPort.receiveFlowFiles(StandardPublicPort.java:561)
> ... 123 common frames omitted
> Caused by: java.lang.InterruptedException: null
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:460)
> at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348)
> at org.apache.nifi.remote.StandardPublicPort$FlowFileRequest.waitForService(StandardPublicPort.java:490)
> at org.apache.nifi.remote.StandardPublicPort.receiveFlowFiles(StandardPublicPort.java:552)
> ... 123 common frames omitted{noformat}
> The Input threads stuck until we restart nifi. Otherwise we cant receive any data over this input anymore.
> Additional Info: We send Logs and stuff with aprox. 80 Minifis (Java) to Nifi. The payloads are between some bytes and ~ 10 MB.
> I think its related to this issue, because before 1.13 we didnt face this problem.
> https://issues.apache.org/jira/browse/NIFI-7566
--
This message was sent by Atlassian Jira
(v8.3.4#803005)