Increasing currentTask to 20 in processor ends up with blocked threads in 5 min, which results in WebUI SocketTimeOutException

[Seokwon Yang <il...@hotmail.com>]

I have a simple flow with 2 processors connected.

The first one is GenerateFlowFile generating 64KB at 10 batch. and second one is PutAzureBlobStorage to send flowfiles generated from the first processor to Azure Blob.

When I increase the currentTask to 20 and leaves the flow, eventually, I see SocketTimeoutException in UI.
And when I capture the thread dump with n='nifi.sh dump', I see 31 blocked threads. For example,:

hreadId:1428 - state:BLOCKED
stackTrace:
at org.apache.nifi.provenance.store.RecordWriterLease.tryClaim(RecordWriterLease.java:49)
at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:178)
at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:103)
at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:180)
at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:766)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:360)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:343)
- waiting on org.apache.nifi.controller.repository.StandardProcessSession@64231af5
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@1246c12d


and:
threadId:3 - state:BLOCKED
stackTrace:
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:139)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)


I have attached the dump file in this post.   I also followed this post (https://community.cloudera.com/t5/Support-Questions/Read-Time-out-issue-with-NiFi-cluster-NiFi-version-nifi-1-2/m-p/231363), but all recommendation didn't solve the problem with the blocked thread with providence store.  Can anyone take a look at my dump file and share insight? Thank you.