You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Manish G <ma...@gmail.com> on 2016/12/01 00:02:38 UTC
Flow File Stuck for no reason
Hi,
I have noticed that sometime a flow file gets stuck on a processor for a
very long time for no reason and then I can not even stop the processor to
look at the flow flow file from queue. If I click on stop, then processor
goes into a state where I cannot start/stop the processor.
On restarting the NiFi, the *file gets processed successfully* and routed
to success queue. I checked in App log, but everything seems to be normal
for the flow file. I don't see anything mysterious in provenance too
(except that queue time is in hours).
Has anyone else faced a similar issue? What else should I check to identify
the root cause for this?
Thanks,
Manish
Re: Flow File Stuck for no reason
Posted by Bryan Bende <bb...@gmail.com>.
Manish,
Ok what I am saying is that NIFI-1922 was simply trying to add one
dependency into the HDFS processors to make the Azure file system
implementations available:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-azure</artifactId>
<version>2.7.2</version>+
</dependency>
In NiFi 1.1 this is no longer necessary because the HDFS processors have a
property where you can point to additional JARS.
I created a directory, lets say /opt/wasb/jars where I put:
azure-storage-2.0.0.jar
commons-codec-1.6.jar
commons-lang3-3.3.2.jar
commons-logging-1.1.1.jar
guava-11.0.2.jar
hadoop-azure-2.7.3.jar
httpclient-4.2.5.jar
httpcore-4.2.4.jar
jackson-core-2.2.3.jar
jsr305-1.3.9.jar
slf4j-api-1.7.5.jar
Then using NiFi 1.1 I created a PutHDFS processor and set the "Additional
Classpath Resources" property to point "/opt/wasb/jars", thus adding all
those JARs to the classpath.
I was able to talk to a WASB filesystem using this approach, as well as an
ADLS filesystem (required a different set of classpath resources).
It might be better trying to run NiFi 1.1 and using the above approach to
see if the problem persists. Right now it is really hard to say since you
are running a custom processor.
Thanks,
Bryan
On Thu, Dec 1, 2016 at 12:35 PM, Manish G <ma...@gmail.com> wrote:
> Bryan,
>
> It's for uploading data on HDInsight cluster and as per the patch in
> Nifi-1922.
>
> Regards,
> Manish
>
>
> On Thu, Dec 1, 2016 at 12:16 PM, Bryan Bende <bb...@gmail.com> wrote:
>
>> Manish,
>>
>> Was the reason for your custom version of PutHDFS just to include the
>> Azure libraries? or was there other functionality you needed to introduce?
>>
>> The reason I'm asking is because in the Apache NiFi 1.1 release (just
>> released) we added an "Additional Classpath Resources" property to all HDFS
>> processors, and I have been able to use that to connect to an Azure WASB
>> filesystem.
>>
>> I can provide the list of JARs I added to the classpath to make it work
>> if that would help.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Thu, Dec 1, 2016 at 12:11 PM, Manish G <ma...@gmail.com> wrote:
>>
>>> Hi Joe,
>>>
>>> I am not using anything from Nifi 1.0 and its all 0.7. What indicates
>>> the use of 1.0?
>>>
>>> Regards,
>>> Manish
>>>
>>> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt <jo...@gmail.com> wrote:
>>>
>>>> Manish
>>>>
>>>> It also appears from the log output you provided so far that this is a
>>>> combination of nifi 0.7 and 1.0 parts. We do not recommend doing this
>>>> and it is not likely to work.
>>>>
>>>> Please try the flow from a single release line. Apache NiFi 1.1.0 is
>>>> available for use now.
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G <ma...@gmail.com>
>>>> wrote:
>>>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I
>>>> see
>>>> > the issue again.
>>>> >
>>>> > Thanks,
>>>> > Manish
>>>> >
>>>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <jo...@gmail.com> wrote:
>>>> >>
>>>> >> Manish
>>>> >>
>>>> >> When it is in the stuck state can you please run
>>>> >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>>>> >> would aid us in narrowing in on a possible cause.
>>>> >>
>>>> >> Thanks
>>>> >> Joe
>>>> >>
>>>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com>
>>>> wrote:
>>>> >> > Hi Joe,
>>>> >> >
>>>> >> > Here is what I can see in the App Log:
>>>> >> >
>>>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>>>> >> > testing.nifi.processor.hdfs.testingPutHDFS ""
>>>> >> > org.apache.hadoop.fs.azure.AzureException:
>>>> >> > java.util.NoSuchElementException:
>>>> >> > An error occurred while enumerating the result, check the original
>>>> >> > exception
>>>> >> > for details.
>>>> >> > at
>>>> >> >
>>>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>>>> veMetadata(AzureNativeFileSystemStore.java:1930)
>>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>>> >> > at
>>>> >> >
>>>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat
>>>> us(NativeAzureFileSystem.java:1592)
>>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>>>> >> > ~[hadoop-common-2.7.2.jar:na]
>>>> >> > at
>>>> >> >
>>>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing
>>>> PutHDFS.java:260)
>>>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>>>> >> > at
>>>> >> >
>>>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
>>>> ctProcessor.java:27)
>>>> >> > [nifi-api-0.7.0.jar:0.7.0]
>>>> >> > at
>>>> >> >
>>>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>>>> tandardProcessorNode.java:1054)
>>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>>> >> > at
>>>> >> >
>>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>>>> .call(ContinuallyRunProcessorTask.java:136)
>>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>>> >> > at
>>>> >> >
>>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>>>> .call(ContinuallyRunProcessorTask.java:47)
>>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>>> >> > at
>>>> >> >
>>>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>>>> gent$1.run(TimerDrivenSchedulingAgent.java:127)
>>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>>> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>> Source)
>>>> >> > [na:1.8.0_101]
>>>> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>>>> >> > [na:1.8.0_101]
>>>> >> > at
>>>> >> >
>>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>> tureTask.access$301(Unknown
>>>> >> > Source) [na:1.8.0_101]
>>>> >> > at
>>>> >> >
>>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>> tureTask.run(Unknown
>>>> >> > Source) [na:1.8.0_101]
>>>> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>>> Source)
>>>> >> > [na:1.8.0_101]
>>>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>> Source)
>>>> >> > [na:1.8.0_101]
>>>> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>>>> >> > Caused by: java.util.NoSuchElementException: An error occurred
>>>> while
>>>> >> > enumerating the result, check the original exception for details.
>>>> >> > at
>>>> >> >
>>>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>>>> xt(LazySegmentedIterator.java:113)
>>>> >> > ~[azure-storage-2.0.0.jar:na]
>>>> >> > at
>>>> >> >
>>>> >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIter
>>>> ator.hasNext(StorageInterfaceImpl.java:128)
>>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>>> >> > at
>>>> >> >
>>>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>>>> veMetadata(AzureNativeFileSystemStore.java:1909)
>>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>>> >> > ... 15 common frames omitted
>>>> >> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>>>> >> > at
>>>> >> >
>>>> >> > com.microsoft.azure.storage.StorageException.translateFromHt
>>>> tpStatus(StorageException.java:202)
>>>> >> > ~[azure-storage-2.0.0.jar:na]
>>>> >> > at
>>>> >> >
>>>> >> > com.microsoft.azure.storage.StorageException.translateExcept
>>>> ion(StorageException.java:172)
>>>> >> > ~[azure-storage-2.0.0.jar:na]
>>>> >> > at
>>>> >> >
>>>> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWith
>>>> Retry(ExecutionEngine.java:273)
>>>> >> > ~[azure-storage-2.0.0.jar:na]
>>>> >> > at
>>>> >> >
>>>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>>>> xt(LazySegmentedIterator.java:109)
>>>> >> > ~[azure-storage-2.0.0.jar:na]
>>>> >> > ... 17 common frames omitted
>>>> >> > Caused by: java.lang.NullPointerException: null
>>>> >> > at
>>>> >> >
>>>> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWith
>>>> Retry(ExecutionEngine.java:181)
>>>> >> > ~[azure-storage-2.0.0.jar:na]
>>>> >> > ... 18 common frames omitted
>>>> >> >
>>>> >> > Regards,
>>>> >> > Manish
>>>> >> >
>>>> >> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com>
>>>> wrote:
>>>> >> >>
>>>> >> >> Manish
>>>> >> >>
>>>> >> >> Please produce and share the thread dump I mentioned.
>>>> >> >>
>>>> >> >> Thanks
>>>> >> >> Joe
>>>> >> >>
>>>> >> >> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com>
>>>> wrote:
>>>> >> >>>
>>>> >> >>> Hi,
>>>> >> >>>
>>>> >> >>> I don't know why, but this is happening now more frequently.
>>>> Where
>>>> >> >>> should
>>>> >> >>> I look into to find the root cause?
>>>> >> >>>
>>>> >> >>> Thanks,
>>>> >> >>> Manish
>>>> >> >>>
>>>> >> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <
>>>> manish.ggoru@gmail.com>
>>>> >> >>> wrote:
>>>> >> >>>>
>>>> >> >>>> Hi Joe,
>>>> >> >>>>
>>>> >> >>>> Thanks for the quick reply. Yes, the processor keeps running on
>>>> a
>>>> >> >>>> single
>>>> >> >>>> thread (even after stopping). And the number remains there even
>>>> after
>>>> >> >>>> stopping.
>>>> >> >>>> Today, it happened on my customized putHDFS processor. Only
>>>> thing
>>>> >> >>>> different in this processor is - I have added an additional
>>>> attribute
>>>> >> >>>> that
>>>> >> >>>> tells if the processor created the directory while loading the
>>>> file
>>>> >> >>>> on HDFS.
>>>> >> >>>> I don't think this should be the issue though.
>>>> >> >>>>
>>>> >> >>>> Regards,
>>>> >> >>>> Manish
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com>
>>>> wrote:
>>>> >> >>>>>
>>>> >> >>>>> Manish
>>>> >> >>>>>
>>>> >> >>>>> When it is stuck do you see a number in the top right corner
>>>> of the
>>>> >> >>>>> processor? When you stop it does the number remain? That
>>>> number is
>>>> >> >>>>> telling you how many threads are still executing. Which
>>>> processor
>>>> >> >>>>> are
>>>> >> >>>>> we talking about? When it is in the stuck state can you
>>>> please run
>>>> >> >>>>> bin/nifi.sh dump. If you can then share the
>>>> nifi-bootstrap.log that
>>>> >> >>>>> would aid us in narrowing in on a possible cause.
>>>> >> >>>>>
>>>> >> >>>>> Thanks
>>>> >> >>>>> Joe
>>>> >> >>>>>
>>>> >> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <
>>>> manish.ggoru@gmail.com>
>>>> >> >>>>> wrote:
>>>> >> >>>>> >
>>>> >> >>>>> > Hi,
>>>> >> >>>>> >
>>>> >> >>>>> > I have noticed that sometime a flow file gets stuck on a
>>>> processor
>>>> >> >>>>> > for a
>>>> >> >>>>> > very long time for no reason and then I can not even stop the
>>>> >> >>>>> > processor to
>>>> >> >>>>> > look at the flow flow file from queue. If I click on stop,
>>>> then
>>>> >> >>>>> > processor
>>>> >> >>>>> > goes into a state where I cannot start/stop the processor.
>>>> >> >>>>> >
>>>> >> >>>>> > On restarting the NiFi, the file gets processed successfully
>>>> and
>>>> >> >>>>> > routed to
>>>> >> >>>>> > success queue. I checked in App log, but everything seems to
>>>> be
>>>> >> >>>>> > normal for
>>>> >> >>>>> > the flow file. I don't see anything mysterious in provenance
>>>> too
>>>> >> >>>>> > (except
>>>> >> >>>>> > that queue time is in hours).
>>>> >> >>>>> >
>>>> >> >>>>> > Has anyone else faced a similar issue? What else should I
>>>> check to
>>>> >> >>>>> > identify
>>>> >> >>>>> > the root cause for this?
>>>> >> >>>>> >
>>>> >> >>>>> > Thanks,
>>>> >> >>>>> > Manish
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> --
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> With Warm Regards,
>>>> >> >>>> Manish
>>>> >> >>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> --
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> With Warm Regards,
>>>> >> >>> Manish
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> >
>>>> >> >
>>>> >> > With Warm Regards,
>>>> >> > Manish
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> >
>>>> > With Warm Regards,
>>>> > Manish
>>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> *With Warm Regards,*
>>> *Manish*
>>>
>>
>>
>
>
> --
>
>
> *With Warm Regards,*
> *Manish*
>
Re: Flow File Stuck for no reason
Posted by Manish G <ma...@gmail.com>.
Bryan,
It's for uploading data on HDInsight cluster and as per the patch in
Nifi-1922.
Regards,
Manish
On Thu, Dec 1, 2016 at 12:16 PM, Bryan Bende <bb...@gmail.com> wrote:
> Manish,
>
> Was the reason for your custom version of PutHDFS just to include the
> Azure libraries? or was there other functionality you needed to introduce?
>
> The reason I'm asking is because in the Apache NiFi 1.1 release (just
> released) we added an "Additional Classpath Resources" property to all HDFS
> processors, and I have been able to use that to connect to an Azure WASB
> filesystem.
>
> I can provide the list of JARs I added to the classpath to make it work if
> that would help.
>
> Thanks,
>
> Bryan
>
>
> On Thu, Dec 1, 2016 at 12:11 PM, Manish G <ma...@gmail.com> wrote:
>
>> Hi Joe,
>>
>> I am not using anything from Nifi 1.0 and its all 0.7. What indicates the
>> use of 1.0?
>>
>> Regards,
>> Manish
>>
>> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>>> Manish
>>>
>>> It also appears from the log output you provided so far that this is a
>>> combination of nifi 0.7 and 1.0 parts. We do not recommend doing this
>>> and it is not likely to work.
>>>
>>> Please try the flow from a single release line. Apache NiFi 1.1.0 is
>>> available for use now.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G <ma...@gmail.com> wrote:
>>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I
>>> see
>>> > the issue again.
>>> >
>>> > Thanks,
>>> > Manish
>>> >
>>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <jo...@gmail.com> wrote:
>>> >>
>>> >> Manish
>>> >>
>>> >> When it is in the stuck state can you please run
>>> >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>>> >> would aid us in narrowing in on a possible cause.
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com>
>>> wrote:
>>> >> > Hi Joe,
>>> >> >
>>> >> > Here is what I can see in the App Log:
>>> >> >
>>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS ""
>>> >> > org.apache.hadoop.fs.azure.AzureException:
>>> >> > java.util.NoSuchElementException:
>>> >> > An error occurred while enumerating the result, check the original
>>> >> > exception
>>> >> > for details.
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>>> veMetadata(AzureNativeFileSystemStore.java:1930)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat
>>> us(NativeAzureFileSystem.java:1592)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>>> >> > ~[hadoop-common-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing
>>> PutHDFS.java:260)
>>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
>>> ctProcessor.java:27)
>>> >> > [nifi-api-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>>> tandardProcessorNode.java:1054)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>>> .call(ContinuallyRunProcessorTask.java:136)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>>> .call(ContinuallyRunProcessorTask.java:47)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>>> gent$1.run(TimerDrivenSchedulingAgent.java:127)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>>> >> > [na:1.8.0_101]
>>> >> > at
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.access$301(Unknown
>>> >> > Source) [na:1.8.0_101]
>>> >> > at
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.run(Unknown
>>> >> > Source) [na:1.8.0_101]
>>> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>> Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>> Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>>> >> > Caused by: java.util.NoSuchElementException: An error occurred
>>> while
>>> >> > enumerating the result, check the original exception for details.
>>> >> > at
>>> >> >
>>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>>> xt(LazySegmentedIterator.java:113)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIter
>>> ator.hasNext(StorageInterfaceImpl.java:128)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>>> veMetadata(AzureNativeFileSystemStore.java:1909)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > ... 15 common frames omitted
>>> >> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>>> >> > at
>>> >> >
>>> >> > com.microsoft.azure.storage.StorageException.translateFromHt
>>> tpStatus(StorageException.java:202)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > at
>>> >> >
>>> >> > com.microsoft.azure.storage.StorageException.translateExcept
>>> ion(StorageException.java:172)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > at
>>> >> >
>>> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWith
>>> Retry(ExecutionEngine.java:273)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > at
>>> >> >
>>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>>> xt(LazySegmentedIterator.java:109)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > ... 17 common frames omitted
>>> >> > Caused by: java.lang.NullPointerException: null
>>> >> > at
>>> >> >
>>> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWith
>>> Retry(ExecutionEngine.java:181)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > ... 18 common frames omitted
>>> >> >
>>> >> > Regards,
>>> >> > Manish
>>> >> >
>>> >> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com>
>>> wrote:
>>> >> >>
>>> >> >> Manish
>>> >> >>
>>> >> >> Please produce and share the thread dump I mentioned.
>>> >> >>
>>> >> >> Thanks
>>> >> >> Joe
>>> >> >>
>>> >> >> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
>>> >> >>>
>>> >> >>> Hi,
>>> >> >>>
>>> >> >>> I don't know why, but this is happening now more frequently. Where
>>> >> >>> should
>>> >> >>> I look into to find the root cause?
>>> >> >>>
>>> >> >>> Thanks,
>>> >> >>> Manish
>>> >> >>>
>>> >> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <manish.ggoru@gmail.com
>>> >
>>> >> >>> wrote:
>>> >> >>>>
>>> >> >>>> Hi Joe,
>>> >> >>>>
>>> >> >>>> Thanks for the quick reply. Yes, the processor keeps running on a
>>> >> >>>> single
>>> >> >>>> thread (even after stopping). And the number remains there even
>>> after
>>> >> >>>> stopping.
>>> >> >>>> Today, it happened on my customized putHDFS processor. Only thing
>>> >> >>>> different in this processor is - I have added an additional
>>> attribute
>>> >> >>>> that
>>> >> >>>> tells if the processor created the directory while loading the
>>> file
>>> >> >>>> on HDFS.
>>> >> >>>> I don't think this should be the issue though.
>>> >> >>>>
>>> >> >>>> Regards,
>>> >> >>>> Manish
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com>
>>> wrote:
>>> >> >>>>>
>>> >> >>>>> Manish
>>> >> >>>>>
>>> >> >>>>> When it is stuck do you see a number in the top right corner of
>>> the
>>> >> >>>>> processor? When you stop it does the number remain? That
>>> number is
>>> >> >>>>> telling you how many threads are still executing. Which
>>> processor
>>> >> >>>>> are
>>> >> >>>>> we talking about? When it is in the stuck state can you please
>>> run
>>> >> >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log
>>> that
>>> >> >>>>> would aid us in narrowing in on a possible cause.
>>> >> >>>>>
>>> >> >>>>> Thanks
>>> >> >>>>> Joe
>>> >> >>>>>
>>> >> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <
>>> manish.ggoru@gmail.com>
>>> >> >>>>> wrote:
>>> >> >>>>> >
>>> >> >>>>> > Hi,
>>> >> >>>>> >
>>> >> >>>>> > I have noticed that sometime a flow file gets stuck on a
>>> processor
>>> >> >>>>> > for a
>>> >> >>>>> > very long time for no reason and then I can not even stop the
>>> >> >>>>> > processor to
>>> >> >>>>> > look at the flow flow file from queue. If I click on stop,
>>> then
>>> >> >>>>> > processor
>>> >> >>>>> > goes into a state where I cannot start/stop the processor.
>>> >> >>>>> >
>>> >> >>>>> > On restarting the NiFi, the file gets processed successfully
>>> and
>>> >> >>>>> > routed to
>>> >> >>>>> > success queue. I checked in App log, but everything seems to
>>> be
>>> >> >>>>> > normal for
>>> >> >>>>> > the flow file. I don't see anything mysterious in provenance
>>> too
>>> >> >>>>> > (except
>>> >> >>>>> > that queue time is in hours).
>>> >> >>>>> >
>>> >> >>>>> > Has anyone else faced a similar issue? What else should I
>>> check to
>>> >> >>>>> > identify
>>> >> >>>>> > the root cause for this?
>>> >> >>>>> >
>>> >> >>>>> > Thanks,
>>> >> >>>>> > Manish
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> --
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> With Warm Regards,
>>> >> >>>> Manish
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>>
>>> >> >>>
>>> >> >>> With Warm Regards,
>>> >> >>> Manish
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> >
>>> >> >
>>> >> > With Warm Regards,
>>> >> > Manish
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> >
>>> > With Warm Regards,
>>> > Manish
>>>
>>
>>
>>
>> --
>>
>>
>> *With Warm Regards,*
>> *Manish*
>>
>
>
--
*With Warm Regards,*
*Manish*
Re: Flow File Stuck for no reason
Posted by Joe Witt <jo...@gmail.com>.
manish
my apologies now I see what you mean. It is 0.7.0 nifi and you have a
custom nar that is 1.0.0 which is totally fine.
Bryan's response though is a good one to follow up on.
Thanks
joe
On Thu, Dec 1, 2016 at 9:16 AM, Bryan Bende <bb...@gmail.com> wrote:
> Manish,
>
> Was the reason for your custom version of PutHDFS just to include the Azure
> libraries? or was there other functionality you needed to introduce?
>
> The reason I'm asking is because in the Apache NiFi 1.1 release (just
> released) we added an "Additional Classpath Resources" property to all HDFS
> processors, and I have been able to use that to connect to an Azure WASB
> filesystem.
>
> I can provide the list of JARs I added to the classpath to make it work if
> that would help.
>
> Thanks,
>
> Bryan
>
>
> On Thu, Dec 1, 2016 at 12:11 PM, Manish G <ma...@gmail.com> wrote:
>>
>> Hi Joe,
>>
>> I am not using anything from Nifi 1.0 and its all 0.7. What indicates the
>> use of 1.0?
>>
>> Regards,
>> Manish
>>
>> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt <jo...@gmail.com> wrote:
>>>
>>> Manish
>>>
>>> It also appears from the log output you provided so far that this is a
>>> combination of nifi 0.7 and 1.0 parts. We do not recommend doing this
>>> and it is not likely to work.
>>>
>>> Please try the flow from a single release line. Apache NiFi 1.1.0 is
>>> available for use now.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G <ma...@gmail.com> wrote:
>>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I
>>> > see
>>> > the issue again.
>>> >
>>> > Thanks,
>>> > Manish
>>> >
>>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <jo...@gmail.com> wrote:
>>> >>
>>> >> Manish
>>> >>
>>> >> When it is in the stuck state can you please run
>>> >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>>> >> would aid us in narrowing in on a possible cause.
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com>
>>> >> wrote:
>>> >> > Hi Joe,
>>> >> >
>>> >> > Here is what I can see in the App Log:
>>> >> >
>>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS ""
>>> >> > org.apache.hadoop.fs.azure.AzureException:
>>> >> > java.util.NoSuchElementException:
>>> >> > An error occurred while enumerating the result, check the original
>>> >> > exception
>>> >> > for details.
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>>> >> > ~[hadoop-common-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
>>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> >> > [nifi-api-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> >> > Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>>> >> > [na:1.8.0_101]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>>> >> > Source) [na:1.8.0_101]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> >> > Source) [na:1.8.0_101]
>>> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>> >> > Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>>> >> > Caused by: java.util.NoSuchElementException: An error occurred while
>>> >> > enumerating the result, check the original exception for details.
>>> >> > at
>>> >> >
>>> >> >
>>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > ... 15 common frames omitted
>>> >> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>>> >> > at
>>> >> >
>>> >> >
>>> >> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > ... 17 common frames omitted
>>> >> > Caused by: java.lang.NullPointerException: null
>>> >> > at
>>> >> >
>>> >> >
>>> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
>>> >> > ~[azure-storage-2.0.0.jar:na]
>>> >> > ... 18 common frames omitted
>>> >> >
>>> >> > Regards,
>>> >> > Manish
>>> >> >
>>> >> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Manish
>>> >> >>
>>> >> >> Please produce and share the thread dump I mentioned.
>>> >> >>
>>> >> >> Thanks
>>> >> >> Joe
>>> >> >>
>>> >> >> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
>>> >> >>>
>>> >> >>> Hi,
>>> >> >>>
>>> >> >>> I don't know why, but this is happening now more frequently. Where
>>> >> >>> should
>>> >> >>> I look into to find the root cause?
>>> >> >>>
>>> >> >>> Thanks,
>>> >> >>> Manish
>>> >> >>>
>>> >> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com>
>>> >> >>> wrote:
>>> >> >>>>
>>> >> >>>> Hi Joe,
>>> >> >>>>
>>> >> >>>> Thanks for the quick reply. Yes, the processor keeps running on a
>>> >> >>>> single
>>> >> >>>> thread (even after stopping). And the number remains there even
>>> >> >>>> after
>>> >> >>>> stopping.
>>> >> >>>> Today, it happened on my customized putHDFS processor. Only thing
>>> >> >>>> different in this processor is - I have added an additional
>>> >> >>>> attribute
>>> >> >>>> that
>>> >> >>>> tells if the processor created the directory while loading the
>>> >> >>>> file
>>> >> >>>> on HDFS.
>>> >> >>>> I don't think this should be the issue though.
>>> >> >>>>
>>> >> >>>> Regards,
>>> >> >>>> Manish
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com>
>>> >> >>>> wrote:
>>> >> >>>>>
>>> >> >>>>> Manish
>>> >> >>>>>
>>> >> >>>>> When it is stuck do you see a number in the top right corner of
>>> >> >>>>> the
>>> >> >>>>> processor? When you stop it does the number remain? That
>>> >> >>>>> number is
>>> >> >>>>> telling you how many threads are still executing. Which
>>> >> >>>>> processor
>>> >> >>>>> are
>>> >> >>>>> we talking about? When it is in the stuck state can you please
>>> >> >>>>> run
>>> >> >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log
>>> >> >>>>> that
>>> >> >>>>> would aid us in narrowing in on a possible cause.
>>> >> >>>>>
>>> >> >>>>> Thanks
>>> >> >>>>> Joe
>>> >> >>>>>
>>> >> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G
>>> >> >>>>> <ma...@gmail.com>
>>> >> >>>>> wrote:
>>> >> >>>>> >
>>> >> >>>>> > Hi,
>>> >> >>>>> >
>>> >> >>>>> > I have noticed that sometime a flow file gets stuck on a
>>> >> >>>>> > processor
>>> >> >>>>> > for a
>>> >> >>>>> > very long time for no reason and then I can not even stop the
>>> >> >>>>> > processor to
>>> >> >>>>> > look at the flow flow file from queue. If I click on stop,
>>> >> >>>>> > then
>>> >> >>>>> > processor
>>> >> >>>>> > goes into a state where I cannot start/stop the processor.
>>> >> >>>>> >
>>> >> >>>>> > On restarting the NiFi, the file gets processed successfully
>>> >> >>>>> > and
>>> >> >>>>> > routed to
>>> >> >>>>> > success queue. I checked in App log, but everything seems to
>>> >> >>>>> > be
>>> >> >>>>> > normal for
>>> >> >>>>> > the flow file. I don't see anything mysterious in provenance
>>> >> >>>>> > too
>>> >> >>>>> > (except
>>> >> >>>>> > that queue time is in hours).
>>> >> >>>>> >
>>> >> >>>>> > Has anyone else faced a similar issue? What else should I
>>> >> >>>>> > check to
>>> >> >>>>> > identify
>>> >> >>>>> > the root cause for this?
>>> >> >>>>> >
>>> >> >>>>> > Thanks,
>>> >> >>>>> > Manish
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> --
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> With Warm Regards,
>>> >> >>>> Manish
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>>
>>> >> >>>
>>> >> >>> With Warm Regards,
>>> >> >>> Manish
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> >
>>> >> >
>>> >> > With Warm Regards,
>>> >> > Manish
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> >
>>> > With Warm Regards,
>>> > Manish
>>
>>
>>
>>
>> --
>>
>>
>> With Warm Regards,
>> Manish
>
>
Re: Flow File Stuck for no reason
Posted by Bryan Bende <bb...@gmail.com>.
Manish,
Was the reason for your custom version of PutHDFS just to include the Azure
libraries? or was there other functionality you needed to introduce?
The reason I'm asking is because in the Apache NiFi 1.1 release (just
released) we added an "Additional Classpath Resources" property to all HDFS
processors, and I have been able to use that to connect to an Azure WASB
filesystem.
I can provide the list of JARs I added to the classpath to make it work if
that would help.
Thanks,
Bryan
On Thu, Dec 1, 2016 at 12:11 PM, Manish G <ma...@gmail.com> wrote:
> Hi Joe,
>
> I am not using anything from Nifi 1.0 and its all 0.7. What indicates the
> use of 1.0?
>
> Regards,
> Manish
>
> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Manish
>>
>> It also appears from the log output you provided so far that this is a
>> combination of nifi 0.7 and 1.0 parts. We do not recommend doing this
>> and it is not likely to work.
>>
>> Please try the flow from a single release line. Apache NiFi 1.1.0 is
>> available for use now.
>>
>> Thanks
>> Joe
>>
>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G <ma...@gmail.com> wrote:
>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I
>> see
>> > the issue again.
>> >
>> > Thanks,
>> > Manish
>> >
>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <jo...@gmail.com> wrote:
>> >>
>> >> Manish
>> >>
>> >> When it is in the stuck state can you please run
>> >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>> >> would aid us in narrowing in on a possible cause.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com>
>> wrote:
>> >> > Hi Joe,
>> >> >
>> >> > Here is what I can see in the App Log:
>> >> >
>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>> >> > testing.nifi.processor.hdfs.testingPutHDFS ""
>> >> > org.apache.hadoop.fs.azure.AzureException:
>> >> > java.util.NoSuchElementException:
>> >> > An error occurred while enumerating the result, check the original
>> >> > exception
>> >> > for details.
>> >> > at
>> >> >
>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>> veMetadata(AzureNativeFileSystemStore.java:1930)
>> >> > ~[hadoop-azure-2.7.2.jar:na]
>> >> > at
>> >> >
>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat
>> us(NativeAzureFileSystem.java:1592)
>> >> > ~[hadoop-azure-2.7.2.jar:na]
>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>> >> > ~[hadoop-common-2.7.2.jar:na]
>> >> > at
>> >> >
>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing
>> PutHDFS.java:260)
>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>> >> > at
>> >> >
>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
>> ctProcessor.java:27)
>> >> > [nifi-api-0.7.0.jar:0.7.0]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>> tandardProcessorNode.java:1054)
>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:136)
>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:47)
>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>> gent$1.run(TimerDrivenSchedulingAgent.java:127)
>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source)
>> >> > [na:1.8.0_101]
>> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>> >> > [na:1.8.0_101]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(Unknown
>> >> > Source) [na:1.8.0_101]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(Unknown
>> >> > Source) [na:1.8.0_101]
>> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> >> > [na:1.8.0_101]
>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> Source)
>> >> > [na:1.8.0_101]
>> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>> >> > Caused by: java.util.NoSuchElementException: An error occurred while
>> >> > enumerating the result, check the original exception for details.
>> >> > at
>> >> >
>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>> xt(LazySegmentedIterator.java:113)
>> >> > ~[azure-storage-2.0.0.jar:na]
>> >> > at
>> >> >
>> >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIter
>> ator.hasNext(StorageInterfaceImpl.java:128)
>> >> > ~[hadoop-azure-2.7.2.jar:na]
>> >> > at
>> >> >
>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>> veMetadata(AzureNativeFileSystemStore.java:1909)
>> >> > ~[hadoop-azure-2.7.2.jar:na]
>> >> > ... 15 common frames omitted
>> >> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>> >> > at
>> >> >
>> >> > com.microsoft.azure.storage.StorageException.translateFromHt
>> tpStatus(StorageException.java:202)
>> >> > ~[azure-storage-2.0.0.jar:na]
>> >> > at
>> >> >
>> >> > com.microsoft.azure.storage.StorageException.translateExcept
>> ion(StorageException.java:172)
>> >> > ~[azure-storage-2.0.0.jar:na]
>> >> > at
>> >> >
>> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWith
>> Retry(ExecutionEngine.java:273)
>> >> > ~[azure-storage-2.0.0.jar:na]
>> >> > at
>> >> >
>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>> xt(LazySegmentedIterator.java:109)
>> >> > ~[azure-storage-2.0.0.jar:na]
>> >> > ... 17 common frames omitted
>> >> > Caused by: java.lang.NullPointerException: null
>> >> > at
>> >> >
>> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWith
>> Retry(ExecutionEngine.java:181)
>> >> > ~[azure-storage-2.0.0.jar:na]
>> >> > ... 18 common frames omitted
>> >> >
>> >> > Regards,
>> >> > Manish
>> >> >
>> >> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com>
>> wrote:
>> >> >>
>> >> >> Manish
>> >> >>
>> >> >> Please produce and share the thread dump I mentioned.
>> >> >>
>> >> >> Thanks
>> >> >> Joe
>> >> >>
>> >> >> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> I don't know why, but this is happening now more frequently. Where
>> >> >>> should
>> >> >>> I look into to find the root cause?
>> >> >>>
>> >> >>> Thanks,
>> >> >>> Manish
>> >> >>>
>> >> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com>
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> Hi Joe,
>> >> >>>>
>> >> >>>> Thanks for the quick reply. Yes, the processor keeps running on a
>> >> >>>> single
>> >> >>>> thread (even after stopping). And the number remains there even
>> after
>> >> >>>> stopping.
>> >> >>>> Today, it happened on my customized putHDFS processor. Only thing
>> >> >>>> different in this processor is - I have added an additional
>> attribute
>> >> >>>> that
>> >> >>>> tells if the processor created the directory while loading the
>> file
>> >> >>>> on HDFS.
>> >> >>>> I don't think this should be the issue though.
>> >> >>>>
>> >> >>>> Regards,
>> >> >>>> Manish
>> >> >>>>
>> >> >>>>
>> >> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com>
>> wrote:
>> >> >>>>>
>> >> >>>>> Manish
>> >> >>>>>
>> >> >>>>> When it is stuck do you see a number in the top right corner of
>> the
>> >> >>>>> processor? When you stop it does the number remain? That
>> number is
>> >> >>>>> telling you how many threads are still executing. Which
>> processor
>> >> >>>>> are
>> >> >>>>> we talking about? When it is in the stuck state can you please
>> run
>> >> >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log
>> that
>> >> >>>>> would aid us in narrowing in on a possible cause.
>> >> >>>>>
>> >> >>>>> Thanks
>> >> >>>>> Joe
>> >> >>>>>
>> >> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <
>> manish.ggoru@gmail.com>
>> >> >>>>> wrote:
>> >> >>>>> >
>> >> >>>>> > Hi,
>> >> >>>>> >
>> >> >>>>> > I have noticed that sometime a flow file gets stuck on a
>> processor
>> >> >>>>> > for a
>> >> >>>>> > very long time for no reason and then I can not even stop the
>> >> >>>>> > processor to
>> >> >>>>> > look at the flow flow file from queue. If I click on stop, then
>> >> >>>>> > processor
>> >> >>>>> > goes into a state where I cannot start/stop the processor.
>> >> >>>>> >
>> >> >>>>> > On restarting the NiFi, the file gets processed successfully
>> and
>> >> >>>>> > routed to
>> >> >>>>> > success queue. I checked in App log, but everything seems to be
>> >> >>>>> > normal for
>> >> >>>>> > the flow file. I don't see anything mysterious in provenance
>> too
>> >> >>>>> > (except
>> >> >>>>> > that queue time is in hours).
>> >> >>>>> >
>> >> >>>>> > Has anyone else faced a similar issue? What else should I
>> check to
>> >> >>>>> > identify
>> >> >>>>> > the root cause for this?
>> >> >>>>> >
>> >> >>>>> > Thanks,
>> >> >>>>> > Manish
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> --
>> >> >>>>
>> >> >>>>
>> >> >>>> With Warm Regards,
>> >> >>>> Manish
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>>
>> >> >>>
>> >> >>> With Warm Regards,
>> >> >>> Manish
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> >
>> >> > With Warm Regards,
>> >> > Manish
>> >
>> >
>> >
>> >
>> > --
>> >
>> >
>> > With Warm Regards,
>> > Manish
>>
>
>
>
> --
>
>
> *With Warm Regards,*
> *Manish*
>
Re: Flow File Stuck for no reason
Posted by Manish G <ma...@gmail.com>.
Hi Joe,
I am not using anything from Nifi 1.0 and its all 0.7. What indicates the
use of 1.0?
Regards,
Manish
On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt <jo...@gmail.com> wrote:
> Manish
>
> It also appears from the log output you provided so far that this is a
> combination of nifi 0.7 and 1.0 parts. We do not recommend doing this
> and it is not likely to work.
>
> Please try the flow from a single release line. Apache NiFi 1.1.0 is
> available for use now.
>
> Thanks
> Joe
>
> On Thu, Dec 1, 2016 at 9:02 AM, Manish G <ma...@gmail.com> wrote:
> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see
> > the issue again.
> >
> > Thanks,
> > Manish
> >
> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <jo...@gmail.com> wrote:
> >>
> >> Manish
> >>
> >> When it is in the stuck state can you please run
> >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
> >> would aid us in narrowing in on a possible cause.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com>
> wrote:
> >> > Hi Joe,
> >> >
> >> > Here is what I can see in the App Log:
> >> >
> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
> >> > testing.nifi.processor.hdfs.testingPutHDFS ""
> >> > org.apache.hadoop.fs.azure.AzureException:
> >> > java.util.NoSuchElementException:
> >> > An error occurred while enumerating the result, check the original
> >> > exception
> >> > for details.
> >> > at
> >> >
> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.
> retrieveMetadata(AzureNativeFileSystemStore.java:1930)
> >> > ~[hadoop-azure-2.7.2.jar:na]
> >> > at
> >> >
> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(
> NativeAzureFileSystem.java:1592)
> >> > ~[hadoop-azure-2.7.2.jar:na]
> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> >> > ~[hadoop-common-2.7.2.jar:na]
> >> > at
> >> >
> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(
> testingPutHDFS.java:260)
> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
> >> > at
> >> >
> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:27)
> >> > [nifi-api-0.7.0.jar:0.7.0]
> >> > at
> >> >
> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1054)
> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
> >> > at
> >> >
> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:136)
> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
> >> > at
> >> >
> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:47)
> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
> >> > at
> >> >
> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.
> run(TimerDrivenSchedulingAgent.java:127)
> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
> >> > [na:1.8.0_101]
> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> >> > [na:1.8.0_101]
> >> > at
> >> >
> >> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(Unknown
> >> > Source) [na:1.8.0_101]
> >> > at
> >> >
> >> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(Unknown
> >> > Source) [na:1.8.0_101]
> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> >> > [na:1.8.0_101]
> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> >> > [na:1.8.0_101]
> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
> >> > Caused by: java.util.NoSuchElementException: An error occurred while
> >> > enumerating the result, check the original exception for details.
> >> > at
> >> >
> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.
> hasNext(LazySegmentedIterator.java:113)
> >> > ~[azure-storage-2.0.0.jar:na]
> >> > at
> >> >
> >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$
> WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
> >> > ~[hadoop-azure-2.7.2.jar:na]
> >> > at
> >> >
> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.
> retrieveMetadata(AzureNativeFileSystemStore.java:1909)
> >> > ~[hadoop-azure-2.7.2.jar:na]
> >> > ... 15 common frames omitted
> >> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
> >> > at
> >> >
> >> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(
> StorageException.java:202)
> >> > ~[azure-storage-2.0.0.jar:na]
> >> > at
> >> >
> >> > com.microsoft.azure.storage.StorageException.translateException(
> StorageException.java:172)
> >> > ~[azure-storage-2.0.0.jar:na]
> >> > at
> >> >
> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(
> ExecutionEngine.java:273)
> >> > ~[azure-storage-2.0.0.jar:na]
> >> > at
> >> >
> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.
> hasNext(LazySegmentedIterator.java:109)
> >> > ~[azure-storage-2.0.0.jar:na]
> >> > ... 17 common frames omitted
> >> > Caused by: java.lang.NullPointerException: null
> >> > at
> >> >
> >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(
> ExecutionEngine.java:181)
> >> > ~[azure-storage-2.0.0.jar:na]
> >> > ... 18 common frames omitted
> >> >
> >> > Regards,
> >> > Manish
> >> >
> >> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com> wrote:
> >> >>
> >> >> Manish
> >> >>
> >> >> Please produce and share the thread dump I mentioned.
> >> >>
> >> >> Thanks
> >> >> Joe
> >> >>
> >> >> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> I don't know why, but this is happening now more frequently. Where
> >> >>> should
> >> >>> I look into to find the root cause?
> >> >>>
> >> >>> Thanks,
> >> >>> Manish
> >> >>>
> >> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com>
> >> >>> wrote:
> >> >>>>
> >> >>>> Hi Joe,
> >> >>>>
> >> >>>> Thanks for the quick reply. Yes, the processor keeps running on a
> >> >>>> single
> >> >>>> thread (even after stopping). And the number remains there even
> after
> >> >>>> stopping.
> >> >>>> Today, it happened on my customized putHDFS processor. Only thing
> >> >>>> different in this processor is - I have added an additional
> attribute
> >> >>>> that
> >> >>>> tells if the processor created the directory while loading the file
> >> >>>> on HDFS.
> >> >>>> I don't think this should be the issue though.
> >> >>>>
> >> >>>> Regards,
> >> >>>> Manish
> >> >>>>
> >> >>>>
> >> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com>
> wrote:
> >> >>>>>
> >> >>>>> Manish
> >> >>>>>
> >> >>>>> When it is stuck do you see a number in the top right corner of
> the
> >> >>>>> processor? When you stop it does the number remain? That number
> is
> >> >>>>> telling you how many threads are still executing. Which processor
> >> >>>>> are
> >> >>>>> we talking about? When it is in the stuck state can you please
> run
> >> >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log
> that
> >> >>>>> would aid us in narrowing in on a possible cause.
> >> >>>>>
> >> >>>>> Thanks
> >> >>>>> Joe
> >> >>>>>
> >> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <manish.ggoru@gmail.com
> >
> >> >>>>> wrote:
> >> >>>>> >
> >> >>>>> > Hi,
> >> >>>>> >
> >> >>>>> > I have noticed that sometime a flow file gets stuck on a
> processor
> >> >>>>> > for a
> >> >>>>> > very long time for no reason and then I can not even stop the
> >> >>>>> > processor to
> >> >>>>> > look at the flow flow file from queue. If I click on stop, then
> >> >>>>> > processor
> >> >>>>> > goes into a state where I cannot start/stop the processor.
> >> >>>>> >
> >> >>>>> > On restarting the NiFi, the file gets processed successfully and
> >> >>>>> > routed to
> >> >>>>> > success queue. I checked in App log, but everything seems to be
> >> >>>>> > normal for
> >> >>>>> > the flow file. I don't see anything mysterious in provenance too
> >> >>>>> > (except
> >> >>>>> > that queue time is in hours).
> >> >>>>> >
> >> >>>>> > Has anyone else faced a similar issue? What else should I check
> to
> >> >>>>> > identify
> >> >>>>> > the root cause for this?
> >> >>>>> >
> >> >>>>> > Thanks,
> >> >>>>> > Manish
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>>
> >> >>>>
> >> >>>> With Warm Regards,
> >> >>>> Manish
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>>
> >> >>>
> >> >>> With Warm Regards,
> >> >>> Manish
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> >
> >> > With Warm Regards,
> >> > Manish
> >
> >
> >
> >
> > --
> >
> >
> > With Warm Regards,
> > Manish
>
--
*With Warm Regards,*
*Manish*
Re: Flow File Stuck for no reason
Posted by Joe Witt <jo...@gmail.com>.
Manish
It also appears from the log output you provided so far that this is a
combination of nifi 0.7 and 1.0 parts. We do not recommend doing this
and it is not likely to work.
Please try the flow from a single release line. Apache NiFi 1.1.0 is
available for use now.
Thanks
Joe
On Thu, Dec 1, 2016 at 9:02 AM, Manish G <ma...@gmail.com> wrote:
> Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see
> the issue again.
>
> Thanks,
> Manish
>
> On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Manish
>>
>> When it is in the stuck state can you please run
>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>> would aid us in narrowing in on a possible cause.
>>
>> Thanks
>> Joe
>>
>> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com> wrote:
>> > Hi Joe,
>> >
>> > Here is what I can see in the App Log:
>> >
>> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>> > testing.nifi.processor.hdfs.testingPutHDFS ""
>> > org.apache.hadoop.fs.azure.AzureException:
>> > java.util.NoSuchElementException:
>> > An error occurred while enumerating the result, check the original
>> > exception
>> > for details.
>> > at
>> >
>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at
>> >
>> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>> > ~[hadoop-common-2.7.2.jar:na]
>> > at
>> >
>> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
>> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>> > at
>> >
>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> > [nifi-api-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>> > [na:1.8.0_101]
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>> > Source) [na:1.8.0_101]
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> > Source) [na:1.8.0_101]
>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>> > Caused by: java.util.NoSuchElementException: An error occurred while
>> > enumerating the result, check the original exception for details.
>> > at
>> >
>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at
>> >
>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > ... 15 common frames omitted
>> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>> > at
>> >
>> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
>> > ~[azure-storage-2.0.0.jar:na]
>> > ... 17 common frames omitted
>> > Caused by: java.lang.NullPointerException: null
>> > at
>> >
>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
>> > ~[azure-storage-2.0.0.jar:na]
>> > ... 18 common frames omitted
>> >
>> > Regards,
>> > Manish
>> >
>> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com> wrote:
>> >>
>> >> Manish
>> >>
>> >> Please produce and share the thread dump I mentioned.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I don't know why, but this is happening now more frequently. Where
>> >>> should
>> >>> I look into to find the root cause?
>> >>>
>> >>> Thanks,
>> >>> Manish
>> >>>
>> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi Joe,
>> >>>>
>> >>>> Thanks for the quick reply. Yes, the processor keeps running on a
>> >>>> single
>> >>>> thread (even after stopping). And the number remains there even after
>> >>>> stopping.
>> >>>> Today, it happened on my customized putHDFS processor. Only thing
>> >>>> different in this processor is - I have added an additional attribute
>> >>>> that
>> >>>> tells if the processor created the directory while loading the file
>> >>>> on HDFS.
>> >>>> I don't think this should be the issue though.
>> >>>>
>> >>>> Regards,
>> >>>> Manish
>> >>>>
>> >>>>
>> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com> wrote:
>> >>>>>
>> >>>>> Manish
>> >>>>>
>> >>>>> When it is stuck do you see a number in the top right corner of the
>> >>>>> processor? When you stop it does the number remain? That number is
>> >>>>> telling you how many threads are still executing. Which processor
>> >>>>> are
>> >>>>> we talking about? When it is in the stuck state can you please run
>> >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>> >>>>> would aid us in narrowing in on a possible cause.
>> >>>>>
>> >>>>> Thanks
>> >>>>> Joe
>> >>>>>
>> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com>
>> >>>>> wrote:
>> >>>>> >
>> >>>>> > Hi,
>> >>>>> >
>> >>>>> > I have noticed that sometime a flow file gets stuck on a processor
>> >>>>> > for a
>> >>>>> > very long time for no reason and then I can not even stop the
>> >>>>> > processor to
>> >>>>> > look at the flow flow file from queue. If I click on stop, then
>> >>>>> > processor
>> >>>>> > goes into a state where I cannot start/stop the processor.
>> >>>>> >
>> >>>>> > On restarting the NiFi, the file gets processed successfully and
>> >>>>> > routed to
>> >>>>> > success queue. I checked in App log, but everything seems to be
>> >>>>> > normal for
>> >>>>> > the flow file. I don't see anything mysterious in provenance too
>> >>>>> > (except
>> >>>>> > that queue time is in hours).
>> >>>>> >
>> >>>>> > Has anyone else faced a similar issue? What else should I check to
>> >>>>> > identify
>> >>>>> > the root cause for this?
>> >>>>> >
>> >>>>> > Thanks,
>> >>>>> > Manish
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>>
>> >>>>
>> >>>> With Warm Regards,
>> >>>> Manish
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>>
>> >>>
>> >>> With Warm Regards,
>> >>> Manish
>> >
>> >
>> >
>> >
>> > --
>> >
>> >
>> > With Warm Regards,
>> > Manish
>
>
>
>
> --
>
>
> With Warm Regards,
> Manish
Re: Flow File Stuck for no reason
Posted by Joe Witt <jo...@gmail.com>.
Hello
This part of the stack dump is the only thing that appears out of
place to me. it appears to be attempting to read from the socket or
stuck there. Perhaps you can use a more recent client library for
that custom HD processor?
"Timer-Driven Process Thread-8" Id=78 RUNNABLE (in native code)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- waiting on java.io.BufferedInputStream@7b23ac8
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
- waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
- waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:124)
at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
at org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem.conditionalRedoFolderRename(NativeAzureFileSystem.java:1638)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1603)
at custom.nifi.processor.hdfs.PutHDFSHDCustom.onTrigger(PutHDFSHDCustom.java:247)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@3b501d78
Thanks
Joe
On Tue, Dec 6, 2016 at 9:05 AM, Manish G <ma...@gmail.com> wrote:
> Hi,
>
> The issue happened again today. I restarted the NiFi and it started working
> again. Below are the rows from bootstrap.log after nifi.sh dump.
> Could this be because of some other process like diagnostic monitor on Azure
> storage blob?
>
> Thanks,
> Manish
>
>
> 2016-12-06 08:41:47,079 INFO [main] org.apache.nifi.bootstrap.RunNiFi "
> "Attach Listener" Id=6 RUNNABLE
>
> "AzureNativeFilesystemStore-UploadBandwidthUpdater" Id=94 TIMED_WAITING on
> null
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.azure.metrics.BandwidthGaugeUpdater$UploadBandwidthUpdater.run(BandwidthGaugeUpdater.java:266)
> at java.lang.Thread.run(Unknown Source)
>
> "Cleanup Archive for default" Id=54 RUNNABLE
> at sun.nio.fs.WindowsNativeDispatcher.FindClose(Native Method)
> at sun.nio.fs.WindowsDirectoryStream.close(Unknown Source)
> at sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(Unknown Source)
> at sun.nio.fs.WindowsFileSystemProvider.checkAccess(Unknown Source)
> at java.nio.file.Files.exists(Unknown Source)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.destroyExpiredArchives(FileSystemRepository.java:1307)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.access$1800(FileSystemRepository.java:84)
> at
> org.apache.nifi.controller.repository.FileSystemRepository$DestroyExpiredArchiveClaims.run(FileSystemRepository.java:1546)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@3cd89c72
>
> "DestroyJavaVM" Id=99 RUNNABLE
>
> "Event-Driven Process Thread-1" Id=56 TIMED_WAITING on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@641ab175
>
> "Event-Driven Process Thread-2" Id=57 TIMED_WAITING on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@3bc33ad9
>
> "Event-Driven Process Thread-3" Id=58 TIMED_WAITING on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@6b35b0f4
>
> "Event-Driven Process Thread-4" Id=59 TIMED_WAITING on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@2cf669f2
>
> "Event-Driven Process Thread-5" Id=60 TIMED_WAITING on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@1dd6e400
>
> "FileSystemRepository Workers Thread-1" Id=52 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "FileSystemRepository Workers Thread-2" Id=53 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "FileSystemRepository Workers Thread-3" Id=65 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "FileSystemRepository Workers Thread-4" Id=66 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@12de219
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(Unknown Source)
> at java.lang.ref.ReferenceQueue.remove(Unknown Source)
> at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
>
> "Flow Service Tasks Thread-1" Id=63 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cbdc2d2
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Flow Service Tasks Thread-2" Id=64 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cbdc2d2
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Framework Task Thread Thread-1" Id=61 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Framework Task Thread Thread-2" Id=100 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Framework Task Thread Thread-3" Id=101 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Framework Task Thread Thread-4" Id=108 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "H2 File Lock Watchdog D:/nififiles/database_repository/nifi-audit.lock.db"
> Id=44 TIMED_WAITING on null
> at java.lang.Thread.sleep(Native Method)
> at org.h2.store.FileLock.run(FileLock.java:517)
> at java.lang.Thread.run(Unknown Source)
>
> "H2 File Lock Watchdog D:/nififiles/database_repository/nifi-users.lock.db"
> Id=42 TIMED_WAITING on null
> at java.lang.Thread.sleep(Native Method)
> at org.h2.store.FileLock.run(FileLock.java:517)
> at java.lang.Thread.run(Unknown Source)
>
> "H2 Log Writer NIFI-AUDIT" Id=45 TIMED_WAITING on
> org.h2.store.WriterThread@3bbc1f86
> at java.lang.Object.wait(Native Method)
> at org.h2.store.WriterThread.run(WriterThread.java:103)
> at java.lang.Thread.run(Unknown Source)
>
> "H2 Log Writer NIFI-USERS" Id=43 TIMED_WAITING on
> org.h2.store.WriterThread@1a7f0896
> at java.lang.Object.wait(Native Method)
> at org.h2.store.WriterThread.run(WriterThread.java:103)
> at java.lang.Thread.run(Unknown Source)
>
> "Listen to Bootstrap" Id=22 RUNNABLE
> at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
> at java.net.PlainSocketImpl.accept(Unknown Source)
> - waiting on java.net.SocksSocketImpl@2d73cb7e
> at java.net.ServerSocket.implAccept(Unknown Source)
> at java.net.ServerSocket.accept(Unknown Source)
> at
> org.apache.nifi.BootstrapListener$Listener.run(BootstrapListener.java:158)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-27-selector-ServerConnectorManager@2086a3b2/0" Id=27
> RUNNABLE (in native code)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> - waiting on sun.nio.ch.Util$2@1b46e697
> - waiting on java.util.Collections$UnmodifiableSet@23c7c131
> - waiting on sun.nio.ch.WindowsSelectorImpl@8bde1da
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
> at
> org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-28-selector-ServerConnectorManager@2086a3b2/1" Id=28
> RUNNABLE (in native code)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> - waiting on sun.nio.ch.Util$2@20930553
> - waiting on java.util.Collections$UnmodifiableSet@7dfac566
> - waiting on sun.nio.ch.WindowsSelectorImpl@472c48e3
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
> at
> org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-29-selector-ServerConnectorManager@2086a3b2/2" Id=29
> RUNNABLE (in native code)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> - waiting on sun.nio.ch.Util$2@6f765847
> - waiting on java.util.Collections$UnmodifiableSet@681a4ed4
> - waiting on sun.nio.ch.WindowsSelectorImpl@48f28891
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
> at
> org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-30-selector-ServerConnectorManager@2086a3b2/3" Id=30
> RUNNABLE (in native code)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> - waiting on sun.nio.ch.Util$2@5fab180d
> - waiting on java.util.Collections$UnmodifiableSet@31922b45
> - waiting on sun.nio.ch.WindowsSelectorImpl@54a785d1
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
> at
> org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web
> Server-31-acceptor-0@236f25b5-ServerConnector@403230be{HTTP/1.1}{0.0.0.0:8080}"
> Id=31 RUNNABLE (in native code)
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
> - waiting on java.lang.Object@5c7d5af
> at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
> - waiting on java.lang.Object@5c7d5af
> at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web
> Server-32-acceptor-1@1657bbfa-ServerConnector@403230be{HTTP/1.1}{0.0.0.0:8080}"
> Id=32 BLOCKED on java.lang.Object@5c7d5af
> at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
> at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-43974" Id=43974 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61cb9802
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-43979" Id=43979 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61cb9802
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-43987" Id=43987 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61cb9802
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@36920bd6Timer" Id=35
> TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7192984e
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@3a7e365Timer" Id=40
> TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1a655aac
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@3c9c6245Timer" Id=38
> TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57f47aa1
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@45adc393Timer" Id=36
> TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@52d7f52
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@5853431fTimer" Id=97
> TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40d96d76
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@6c90063fTimer" Id=98
> TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542a5a90
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@71adfeddTimer" Id=39
> TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c6f1485
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@7c663eafTimer" Id=37
> TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35ade371
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@b6182cbTimer" Id=95
> TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@39a0e827
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "pool-1-thread-1" Id=43988 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f0c7b93
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "pool-1-thread-2" Id=43989 RUNNABLE
> at sun.management.ThreadImpl.dumpThreads0(Native Method)
> at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
> at org.apache.nifi.BootstrapListener.writeDump(BootstrapListener.java:221)
> at org.apache.nifi.BootstrapListener.access$500(BootstrapListener.java:49)
> at
> org.apache.nifi.BootstrapListener$Listener$1.run(BootstrapListener.java:196)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@162f0c7
>
> "pool-16-thread-1" Id=69 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47f9a0c5
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Maintenance Thread-1" Id=48 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3995f134
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Maintenance Thread-2" Id=49 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3995f134
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Maintenance Thread-3" Id=50 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3995f134
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Query Thread-1" Id=35981 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@65f410e3
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Repository Rollover Thread-1" Id=116 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20f69239
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Repository Rollover Thread-2" Id=132 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20f69239
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@1cfe6544
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Unknown Source)
> at java.lang.ref.Reference.tryHandlePending(Unknown Source)
> at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
>
> "Scheduler-1822651119" Id=102 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@119c3348
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Signal Dispatcher" Id=5 RUNNABLE
>
> "StandardProcessScheduler Thread-1" Id=73 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-1" Id=67 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-2" Id=82 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-2" Id=68 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-3" Id=84 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-3" Id=72 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-4" Id=86 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-4" Id=81 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-5" Id=88 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-5" Id=83 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-6" Id=90 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-6" Id=85 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-7" Id=91 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-7" Id=87 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-8" Id=92 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-8" Id=89 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer for 'azure-file-system' metrics system" Id=93 TIMED_WAITING on
> java.util.TaskQueue@49d69a28
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Unknown Source)
> at java.util.TimerThread.run(Unknown Source)
>
> "Timer-Driven Process Thread-1" Id=62 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-10" Id=80 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-2" Id=70 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-3" Id=71 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-4" Id=74 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-5" Id=75 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-6" Id=76 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-7" Id=77 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-8" Id=78 RUNNABLE (in native code)
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(Unknown Source)
> at java.net.SocketInputStream.read(Unknown Source)
> at java.net.SocketInputStream.read(Unknown Source)
> at java.io.BufferedInputStream.fill(Unknown Source)
> at java.io.BufferedInputStream.read1(Unknown Source)
> at java.io.BufferedInputStream.read(Unknown Source)
> - waiting on java.io.BufferedInputStream@7b23ac8
> at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
> at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown
> Source)
> - waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown
> Source)
> - waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
> at java.net.HttpURLConnection.getResponseCode(Unknown Source)
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:124)
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
> at
> org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.conditionalRedoFolderRename(NativeAzureFileSystem.java:1638)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1603)
> at
> custom.nifi.processor.hdfs.PutHDFSHDCustom.onTrigger(PutHDFSHDCustom.java:247)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@3b501d78
>
> "Timer-Driven Process Thread-9" Id=79 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Write-Ahead Local State Provider Maintenance" Id=55 TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3dc7abae
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> "
>
>
> On Thu, Dec 1, 2016 at 12:02 PM, Manish G <ma...@gmail.com> wrote:
>>
>> Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see
>> the issue again.
>>
>> Thanks,
>> Manish
>>
>> On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <jo...@gmail.com> wrote:
>>>
>>> Manish
>>>
>>> When it is in the stuck state can you please run
>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>>> would aid us in narrowing in on a possible cause.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com> wrote:
>>> > Hi Joe,
>>> >
>>> > Here is what I can see in the App Log:
>>> >
>>> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>>> > testing.nifi.processor.hdfs.testingPutHDFS ""
>>> > org.apache.hadoop.fs.azure.AzureException:
>>> > java.util.NoSuchElementException:
>>> > An error occurred while enumerating the result, check the original
>>> > exception
>>> > for details.
>>> > at
>>> >
>>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
>>> > ~[hadoop-azure-2.7.2.jar:na]
>>> > at
>>> >
>>> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
>>> > ~[hadoop-azure-2.7.2.jar:na]
>>> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>>> > ~[hadoop-common-2.7.2.jar:na]
>>> > at
>>> >
>>> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
>>> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>>> > at
>>> >
>>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> > [nifi-api-0.7.0.jar:0.7.0]
>>> > at
>>> >
>>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> > at
>>> >
>>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> > at
>>> >
>>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> > at
>>> >
>>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
>>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>> > [na:1.8.0_101]
>>> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>>> > [na:1.8.0_101]
>>> > at
>>> >
>>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>>> > Source) [na:1.8.0_101]
>>> > at
>>> >
>>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> > Source) [na:1.8.0_101]
>>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>> > [na:1.8.0_101]
>>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>> > [na:1.8.0_101]
>>> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>>> > Caused by: java.util.NoSuchElementException: An error occurred while
>>> > enumerating the result, check the original exception for details.
>>> > at
>>> >
>>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > at
>>> >
>>> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
>>> > ~[hadoop-azure-2.7.2.jar:na]
>>> > at
>>> >
>>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
>>> > ~[hadoop-azure-2.7.2.jar:na]
>>> > ... 15 common frames omitted
>>> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>>> > at
>>> >
>>> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > at
>>> >
>>> > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > at
>>> >
>>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > at
>>> >
>>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > ... 17 common frames omitted
>>> > Caused by: java.lang.NullPointerException: null
>>> > at
>>> >
>>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > ... 18 common frames omitted
>>> >
>>> > Regards,
>>> > Manish
>>> >
>>> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com> wrote:
>>> >>
>>> >> Manish
>>> >>
>>> >> Please produce and share the thread dump I mentioned.
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>> >> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
>>> >>>
>>> >>> Hi,
>>> >>>
>>> >>> I don't know why, but this is happening now more frequently. Where
>>> >>> should
>>> >>> I look into to find the root cause?
>>> >>>
>>> >>> Thanks,
>>> >>> Manish
>>> >>>
>>> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> Hi Joe,
>>> >>>>
>>> >>>> Thanks for the quick reply. Yes, the processor keeps running on a
>>> >>>> single
>>> >>>> thread (even after stopping). And the number remains there even
>>> >>>> after
>>> >>>> stopping.
>>> >>>> Today, it happened on my customized putHDFS processor. Only thing
>>> >>>> different in this processor is - I have added an additional
>>> >>>> attribute that
>>> >>>> tells if the processor created the directory while loading the file
>>> >>>> on HDFS.
>>> >>>> I don't think this should be the issue though.
>>> >>>>
>>> >>>> Regards,
>>> >>>> Manish
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Manish
>>> >>>>>
>>> >>>>> When it is stuck do you see a number in the top right corner of the
>>> >>>>> processor? When you stop it does the number remain? That number
>>> >>>>> is
>>> >>>>> telling you how many threads are still executing. Which processor
>>> >>>>> are
>>> >>>>> we talking about? When it is in the stuck state can you please run
>>> >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log
>>> >>>>> that
>>> >>>>> would aid us in narrowing in on a possible cause.
>>> >>>>>
>>> >>>>> Thanks
>>> >>>>> Joe
>>> >>>>>
>>> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com>
>>> >>>>> wrote:
>>> >>>>> >
>>> >>>>> > Hi,
>>> >>>>> >
>>> >>>>> > I have noticed that sometime a flow file gets stuck on a
>>> >>>>> > processor
>>> >>>>> > for a
>>> >>>>> > very long time for no reason and then I can not even stop the
>>> >>>>> > processor to
>>> >>>>> > look at the flow flow file from queue. If I click on stop, then
>>> >>>>> > processor
>>> >>>>> > goes into a state where I cannot start/stop the processor.
>>> >>>>> >
>>> >>>>> > On restarting the NiFi, the file gets processed successfully and
>>> >>>>> > routed to
>>> >>>>> > success queue. I checked in App log, but everything seems to be
>>> >>>>> > normal for
>>> >>>>> > the flow file. I don't see anything mysterious in provenance too
>>> >>>>> > (except
>>> >>>>> > that queue time is in hours).
>>> >>>>> >
>>> >>>>> > Has anyone else faced a similar issue? What else should I check
>>> >>>>> > to
>>> >>>>> > identify
>>> >>>>> > the root cause for this?
>>> >>>>> >
>>> >>>>> > Thanks,
>>> >>>>> > Manish
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>>
>>> >>>>
>>> >>>> With Warm Regards,
>>> >>>> Manish
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>>
>>> >>>
>>> >>> With Warm Regards,
>>> >>> Manish
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> >
>>> > With Warm Regards,
>>> > Manish
>>
>>
>>
>>
>> --
>>
>>
>> With Warm Regards,
>> Manish
>
>
>
>
> --
>
>
> With Warm Regards,
> Manish
Re: Flow File Stuck for no reason
Posted by Manish G <ma...@gmail.com>.
Hi,
The issue happened again today. I restarted the NiFi and it started working
again. Below are the rows from bootstrap.log after nifi.sh dump.
Could this be because of some other process like diagnostic monitor on
Azure storage blob?
Thanks,
Manish
2016-12-06 08:41:47,079 INFO [main] org.apache.nifi.bootstrap.RunNiFi "
"Attach Listener" Id=6 RUNNABLE
"AzureNativeFilesystemStore-UploadBandwidthUpdater" Id=94 TIMED_WAITING on
null
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.fs.azure.metrics.BandwidthGaugeUpdater$UploadBandwidthUpdater.run(BandwidthGaugeUpdater.java:266)
at java.lang.Thread.run(Unknown Source)
"Cleanup Archive for default" Id=54 RUNNABLE
at sun.nio.fs.WindowsNativeDispatcher.FindClose(Native Method)
at sun.nio.fs.WindowsDirectoryStream.close(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.checkAccess(Unknown Source)
at java.nio.file.Files.exists(Unknown Source)
at
org.apache.nifi.controller.repository.FileSystemRepository.destroyExpiredArchives(FileSystemRepository.java:1307)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1800(FileSystemRepository.java:84)
at
org.apache.nifi.controller.repository.FileSystemRepository$DestroyExpiredArchiveClaims.run(FileSystemRepository.java:1546)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@3cd89c72
"DestroyJavaVM" Id=99 RUNNABLE
"Event-Driven Process Thread-1" Id=56 TIMED_WAITING on
java.lang.Object@37b2da15
at java.lang.Object.wait(Native Method)
at
org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
at
org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@641ab175
"Event-Driven Process Thread-2" Id=57 TIMED_WAITING on
java.lang.Object@37b2da15
at java.lang.Object.wait(Native Method)
at
org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
at
org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@3bc33ad9
"Event-Driven Process Thread-3" Id=58 TIMED_WAITING on
java.lang.Object@37b2da15
at java.lang.Object.wait(Native Method)
at
org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
at
org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@6b35b0f4
"Event-Driven Process Thread-4" Id=59 TIMED_WAITING on
java.lang.Object@37b2da15
at java.lang.Object.wait(Native Method)
at
org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
at
org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@2cf669f2
"Event-Driven Process Thread-5" Id=60 TIMED_WAITING on
java.lang.Object@37b2da15
at java.lang.Object.wait(Native Method)
at
org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
at
org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@1dd6e400
"FileSystemRepository Workers Thread-1" Id=52 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"FileSystemRepository Workers Thread-2" Id=53 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"FileSystemRepository Workers Thread-3" Id=65 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"FileSystemRepository Workers Thread-4" Id=66 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@12de219
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
"Flow Service Tasks Thread-1" Id=63 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cbdc2d2
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Flow Service Tasks Thread-2" Id=64 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cbdc2d2
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Framework Task Thread Thread-1" Id=61 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Framework Task Thread Thread-2" Id=100 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Framework Task Thread Thread-3" Id=101 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Framework Task Thread Thread-4" Id=108 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"H2 File Lock Watchdog D:/nififiles/database_repository/nifi-audit.lock.db"
Id=44 TIMED_WAITING on null
at java.lang.Thread.sleep(Native Method)
at org.h2.store.FileLock.run(FileLock.java:517)
at java.lang.Thread.run(Unknown Source)
"H2 File Lock Watchdog D:/nififiles/database_repository/nifi-users.lock.db"
Id=42 TIMED_WAITING on null
at java.lang.Thread.sleep(Native Method)
at org.h2.store.FileLock.run(FileLock.java:517)
at java.lang.Thread.run(Unknown Source)
"H2 Log Writer NIFI-AUDIT" Id=45 TIMED_WAITING on
org.h2.store.WriterThread@3bbc1f86
at java.lang.Object.wait(Native Method)
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Unknown Source)
"H2 Log Writer NIFI-USERS" Id=43 TIMED_WAITING on
org.h2.store.WriterThread@1a7f0896
at java.lang.Object.wait(Native Method)
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Unknown Source)
"Listen to Bootstrap" Id=22 RUNNABLE
at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.PlainSocketImpl.accept(Unknown Source)
- waiting on java.net.SocksSocketImpl@2d73cb7e
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at
org.apache.nifi.BootstrapListener$Listener.run(BootstrapListener.java:158)
at java.lang.Thread.run(Unknown Source)
"NiFi Web Server-27-selector-ServerConnectorManager@2086a3b2/0" Id=27
RUNNABLE (in native code)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- waiting on sun.nio.ch.Util$2@1b46e697
- waiting on java.util.Collections$UnmodifiableSet@23c7c131
- waiting on sun.nio.ch.WindowsSelectorImpl@8bde1da
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
at
org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
at
org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Unknown Source)
"NiFi Web Server-28-selector-ServerConnectorManager@2086a3b2/1" Id=28
RUNNABLE (in native code)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- waiting on sun.nio.ch.Util$2@20930553
- waiting on java.util.Collections$UnmodifiableSet@7dfac566
- waiting on sun.nio.ch.WindowsSelectorImpl@472c48e3
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
at
org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
at
org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Unknown Source)
"NiFi Web Server-29-selector-ServerConnectorManager@2086a3b2/2" Id=29
RUNNABLE (in native code)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- waiting on sun.nio.ch.Util$2@6f765847
- waiting on java.util.Collections$UnmodifiableSet@681a4ed4
- waiting on sun.nio.ch.WindowsSelectorImpl@48f28891
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
at
org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
at
org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Unknown Source)
"NiFi Web Server-30-selector-ServerConnectorManager@2086a3b2/3" Id=30
RUNNABLE (in native code)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- waiting on sun.nio.ch.Util$2@5fab180d
- waiting on java.util.Collections$UnmodifiableSet@31922b45
- waiting on sun.nio.ch.WindowsSelectorImpl@54a785d1
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
at
org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
at
org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Unknown Source)
"NiFi Web Server-31-acceptor-0@236f25b5-ServerConnector@403230be{HTTP/1.1}{
0.0.0.0:8080}" Id=31 RUNNABLE (in native code)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
- waiting on java.lang.Object@5c7d5af
at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
- waiting on java.lang.Object@5c7d5af
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Unknown Source)
"NiFi Web Server-32-acceptor-1@1657bbfa-ServerConnector@403230be{HTTP/1.1}{
0.0.0.0:8080}" Id=32 BLOCKED on java.lang.Object@5c7d5af
at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Unknown Source)
"NiFi Web Server-43974" Id=43974 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61cb9802
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Unknown Source)
"NiFi Web Server-43979" Id=43979 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61cb9802
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Unknown Source)
"NiFi Web Server-43987" Id=43987 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61cb9802
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Unknown Source)
"org.eclipse.jetty.server.session.HashSessionManager@36920bd6Timer" Id=35
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7192984e
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"org.eclipse.jetty.server.session.HashSessionManager@3a7e365Timer" Id=40
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1a655aac
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"org.eclipse.jetty.server.session.HashSessionManager@3c9c6245Timer" Id=38
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57f47aa1
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"org.eclipse.jetty.server.session.HashSessionManager@45adc393Timer" Id=36
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@52d7f52
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"org.eclipse.jetty.server.session.HashSessionManager@5853431fTimer" Id=97
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40d96d76
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"org.eclipse.jetty.server.session.HashSessionManager@6c90063fTimer" Id=98
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542a5a90
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"org.eclipse.jetty.server.session.HashSessionManager@71adfeddTimer" Id=39
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c6f1485
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"org.eclipse.jetty.server.session.HashSessionManager@7c663eafTimer" Id=37
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35ade371
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"org.eclipse.jetty.server.session.HashSessionManager@b6182cbTimer" Id=95
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@39a0e827
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"pool-1-thread-1" Id=43988 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f0c7b93
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"pool-1-thread-2" Id=43989 RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at org.apache.nifi.BootstrapListener.writeDump(BootstrapListener.java:221)
at org.apache.nifi.BootstrapListener.access$500(BootstrapListener.java:49)
at
org.apache.nifi.BootstrapListener$Listener$1.run(BootstrapListener.java:196)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@162f0c7
"pool-16-thread-1" Id=69 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47f9a0c5
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Provenance Maintenance Thread-1" Id=48 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3995f134
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Provenance Maintenance Thread-2" Id=49 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3995f134
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Provenance Maintenance Thread-3" Id=50 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3995f134
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Provenance Query Thread-1" Id=35981 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@65f410e3
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Provenance Repository Rollover Thread-1" Id=116 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20f69239
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Provenance Repository Rollover Thread-2" Id=132 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20f69239
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@1cfe6544
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at java.lang.ref.Reference.tryHandlePending(Unknown Source)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
"Scheduler-1822651119" Id=102 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@119c3348
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Signal Dispatcher" Id=5 RUNNABLE
"StandardProcessScheduler Thread-1" Id=73 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-1" Id=67 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-2" Id=82 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-2" Id=68 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-3" Id=84 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-3" Id=72 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-4" Id=86 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-4" Id=81 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-5" Id=88 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-5" Id=83 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-6" Id=90 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-6" Id=85 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-7" Id=91 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-7" Id=87 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-8" Id=92 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"StandardProcessScheduler Thread-8" Id=89 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer for 'azure-file-system' metrics system" Id=93 TIMED_WAITING on
java.util.TaskQueue@49d69a28
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
"Timer-Driven Process Thread-1" Id=62 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer-Driven Process Thread-10" Id=80 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer-Driven Process Thread-2" Id=70 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer-Driven Process Thread-3" Id=71 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer-Driven Process Thread-4" Id=74 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer-Driven Process Thread-5" Id=75 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer-Driven Process Thread-6" Id=76 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer-Driven Process Thread-7" Id=77 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer-Driven Process Thread-8" Id=78 RUNNABLE (in native code)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- waiting on java.io.BufferedInputStream@7b23ac8
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown
Source)
- waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown
Source)
- waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:124)
at
com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
at
org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.conditionalRedoFolderRename(NativeAzureFileSystem.java:1638)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1603)
at
custom.nifi.processor.hdfs.PutHDFSHDCustom.onTrigger(PutHDFSHDCustom.java:247)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@3b501d78
"Timer-Driven Process Thread-9" Id=79 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Write-Ahead Local State Provider Maintenance" Id=55 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3dc7abae
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"
On Thu, Dec 1, 2016 at 12:02 PM, Manish G <ma...@gmail.com> wrote:
> Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see
> the issue again.
>
> Thanks,
> Manish
>
> On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <jo...@gmail.com> wrote:
>
>> Manish
>>
>> When it is in the stuck state can you please run
>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>> would aid us in narrowing in on a possible cause.
>>
>> Thanks
>> Joe
>>
>> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com> wrote:
>> > Hi Joe,
>> >
>> > Here is what I can see in the App Log:
>> >
>> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>> > testing.nifi.processor.hdfs.testingPutHDFS ""
>> > org.apache.hadoop.fs.azure.AzureException:
>> java.util.NoSuchElementException:
>> > An error occurred while enumerating the result, check the original
>> exception
>> > for details.
>> > at
>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>> veMetadata(AzureNativeFileSystemStore.java:1930)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at
>> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat
>> us(NativeAzureFileSystem.java:1592)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>> > ~[hadoop-common-2.7.2.jar:na]
>> > at
>> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing
>> PutHDFS.java:260)
>> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>> > at
>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
>> ctProcessor.java:27)
>> > [nifi-api-0.7.0.jar:0.7.0]
>> > at
>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>> tandardProcessorNode.java:1054)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:136)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:47)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>> gent$1.run(TimerDrivenSchedulingAgent.java:127)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>> > [na:1.8.0_101]
>> > at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(Unknown
>> > Source) [na:1.8.0_101]
>> > at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(Unknown
>> > Source) [na:1.8.0_101]
>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>> > Caused by: java.util.NoSuchElementException: An error occurred while
>> > enumerating the result, check the original exception for details.
>> > at
>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>> xt(LazySegmentedIterator.java:113)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIter
>> ator.hasNext(StorageInterfaceImpl.java:128)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at
>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>> veMetadata(AzureNativeFileSystemStore.java:1909)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > ... 15 common frames omitted
>> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>> > at
>> > com.microsoft.azure.storage.StorageException.translateFromHt
>> tpStatus(StorageException.java:202)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> > com.microsoft.azure.storage.StorageException.translateExcept
>> ion(StorageException.java:172)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWith
>> Retry(ExecutionEngine.java:273)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>> xt(LazySegmentedIterator.java:109)
>> > ~[azure-storage-2.0.0.jar:na]
>> > ... 17 common frames omitted
>> > Caused by: java.lang.NullPointerException: null
>> > at
>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWith
>> Retry(ExecutionEngine.java:181)
>> > ~[azure-storage-2.0.0.jar:na]
>> > ... 18 common frames omitted
>> >
>> > Regards,
>> > Manish
>> >
>> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com> wrote:
>> >>
>> >> Manish
>> >>
>> >> Please produce and share the thread dump I mentioned.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I don't know why, but this is happening now more frequently. Where
>> should
>> >>> I look into to find the root cause?
>> >>>
>> >>> Thanks,
>> >>> Manish
>> >>>
>> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com>
>> wrote:
>> >>>>
>> >>>> Hi Joe,
>> >>>>
>> >>>> Thanks for the quick reply. Yes, the processor keeps running on a
>> single
>> >>>> thread (even after stopping). And the number remains there even after
>> >>>> stopping.
>> >>>> Today, it happened on my customized putHDFS processor. Only thing
>> >>>> different in this processor is - I have added an additional
>> attribute that
>> >>>> tells if the processor created the directory while loading the file
>> on HDFS.
>> >>>> I don't think this should be the issue though.
>> >>>>
>> >>>> Regards,
>> >>>> Manish
>> >>>>
>> >>>>
>> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>> Manish
>> >>>>>
>> >>>>> When it is stuck do you see a number in the top right corner of the
>> >>>>> processor? When you stop it does the number remain? That number is
>> >>>>> telling you how many threads are still executing. Which processor
>> are
>> >>>>> we talking about? When it is in the stuck state can you please run
>> >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>> >>>>> would aid us in narrowing in on a possible cause.
>> >>>>>
>> >>>>> Thanks
>> >>>>> Joe
>> >>>>>
>> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com>
>> >>>>> wrote:
>> >>>>> >
>> >>>>> > Hi,
>> >>>>> >
>> >>>>> > I have noticed that sometime a flow file gets stuck on a processor
>> >>>>> > for a
>> >>>>> > very long time for no reason and then I can not even stop the
>> >>>>> > processor to
>> >>>>> > look at the flow flow file from queue. If I click on stop, then
>> >>>>> > processor
>> >>>>> > goes into a state where I cannot start/stop the processor.
>> >>>>> >
>> >>>>> > On restarting the NiFi, the file gets processed successfully and
>> >>>>> > routed to
>> >>>>> > success queue. I checked in App log, but everything seems to be
>> >>>>> > normal for
>> >>>>> > the flow file. I don't see anything mysterious in provenance too
>> >>>>> > (except
>> >>>>> > that queue time is in hours).
>> >>>>> >
>> >>>>> > Has anyone else faced a similar issue? What else should I check to
>> >>>>> > identify
>> >>>>> > the root cause for this?
>> >>>>> >
>> >>>>> > Thanks,
>> >>>>> > Manish
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>>
>> >>>>
>> >>>> With Warm Regards,
>> >>>> Manish
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>>
>> >>>
>> >>> With Warm Regards,
>> >>> Manish
>> >
>> >
>> >
>> >
>> > --
>> >
>> >
>> > With Warm Regards,
>> > Manish
>>
>
>
>
> --
>
>
> *With Warm Regards,*
> *Manish*
>
--
*With Warm Regards,*
*Manish*
Re: Flow File Stuck for no reason
Posted by Manish G <ma...@gmail.com>.
Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see
the issue again.
Thanks,
Manish
On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <jo...@gmail.com> wrote:
> Manish
>
> When it is in the stuck state can you please run
> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
> would aid us in narrowing in on a possible cause.
>
> Thanks
> Joe
>
> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com> wrote:
> > Hi Joe,
> >
> > Here is what I can see in the App Log:
> >
> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
> > testing.nifi.processor.hdfs.testingPutHDFS ""
> > org.apache.hadoop.fs.azure.AzureException: java.util.
> NoSuchElementException:
> > An error occurred while enumerating the result, check the original
> exception
> > for details.
> > at
> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(
> AzureNativeFileSystemStore.java:1930)
> > ~[hadoop-azure-2.7.2.jar:na]
> > at
> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(
> NativeAzureFileSystem.java:1592)
> > ~[hadoop-azure-2.7.2.jar:na]
> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> > ~[hadoop-common-2.7.2.jar:na]
> > at
> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(
> testingPutHDFS.java:260)
> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
> > at
> > org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:27)
> > [nifi-api-0.7.0.jar:0.7.0]
> > at
> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1054)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:136)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:47)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > at
> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(
> TimerDrivenSchedulingAgent.java:127)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> > [na:1.8.0_101]
> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> > [na:1.8.0_101]
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(Unknown
> > Source) [na:1.8.0_101]
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(Unknown
> > Source) [na:1.8.0_101]
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > [na:1.8.0_101]
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > [na:1.8.0_101]
> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
> > Caused by: java.util.NoSuchElementException: An error occurred while
> > enumerating the result, check the original exception for details.
> > at
> > com.microsoft.azure.storage.core.LazySegmentedIterator.
> hasNext(LazySegmentedIterator.java:113)
> > ~[azure-storage-2.0.0.jar:na]
> > at
> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$
> WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
> > ~[hadoop-azure-2.7.2.jar:na]
> > at
> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(
> AzureNativeFileSystemStore.java:1909)
> > ~[hadoop-azure-2.7.2.jar:na]
> > ... 15 common frames omitted
> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
> > at
> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(
> StorageException.java:202)
> > ~[azure-storage-2.0.0.jar:na]
> > at
> > com.microsoft.azure.storage.StorageException.translateException(
> StorageException.java:172)
> > ~[azure-storage-2.0.0.jar:na]
> > at
> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(
> ExecutionEngine.java:273)
> > ~[azure-storage-2.0.0.jar:na]
> > at
> > com.microsoft.azure.storage.core.LazySegmentedIterator.
> hasNext(LazySegmentedIterator.java:109)
> > ~[azure-storage-2.0.0.jar:na]
> > ... 17 common frames omitted
> > Caused by: java.lang.NullPointerException: null
> > at
> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(
> ExecutionEngine.java:181)
> > ~[azure-storage-2.0.0.jar:na]
> > ... 18 common frames omitted
> >
> > Regards,
> > Manish
> >
> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com> wrote:
> >>
> >> Manish
> >>
> >> Please produce and share the thread dump I mentioned.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I don't know why, but this is happening now more frequently. Where
> should
> >>> I look into to find the root cause?
> >>>
> >>> Thanks,
> >>> Manish
> >>>
> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com>
> wrote:
> >>>>
> >>>> Hi Joe,
> >>>>
> >>>> Thanks for the quick reply. Yes, the processor keeps running on a
> single
> >>>> thread (even after stopping). And the number remains there even after
> >>>> stopping.
> >>>> Today, it happened on my customized putHDFS processor. Only thing
> >>>> different in this processor is - I have added an additional attribute
> that
> >>>> tells if the processor created the directory while loading the file
> on HDFS.
> >>>> I don't think this should be the issue though.
> >>>>
> >>>> Regards,
> >>>> Manish
> >>>>
> >>>>
> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com> wrote:
> >>>>>
> >>>>> Manish
> >>>>>
> >>>>> When it is stuck do you see a number in the top right corner of the
> >>>>> processor? When you stop it does the number remain? That number is
> >>>>> telling you how many threads are still executing. Which processor
> are
> >>>>> we talking about? When it is in the stuck state can you please run
> >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
> >>>>> would aid us in narrowing in on a possible cause.
> >>>>>
> >>>>> Thanks
> >>>>> Joe
> >>>>>
> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com>
> >>>>> wrote:
> >>>>> >
> >>>>> > Hi,
> >>>>> >
> >>>>> > I have noticed that sometime a flow file gets stuck on a processor
> >>>>> > for a
> >>>>> > very long time for no reason and then I can not even stop the
> >>>>> > processor to
> >>>>> > look at the flow flow file from queue. If I click on stop, then
> >>>>> > processor
> >>>>> > goes into a state where I cannot start/stop the processor.
> >>>>> >
> >>>>> > On restarting the NiFi, the file gets processed successfully and
> >>>>> > routed to
> >>>>> > success queue. I checked in App log, but everything seems to be
> >>>>> > normal for
> >>>>> > the flow file. I don't see anything mysterious in provenance too
> >>>>> > (except
> >>>>> > that queue time is in hours).
> >>>>> >
> >>>>> > Has anyone else faced a similar issue? What else should I check to
> >>>>> > identify
> >>>>> > the root cause for this?
> >>>>> >
> >>>>> > Thanks,
> >>>>> > Manish
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>>
> >>>> With Warm Regards,
> >>>> Manish
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>>
> >>> With Warm Regards,
> >>> Manish
> >
> >
> >
> >
> > --
> >
> >
> > With Warm Regards,
> > Manish
>
--
*With Warm Regards,*
*Manish*
Re: Flow File Stuck for no reason
Posted by Joe Witt <jo...@gmail.com>.
Manish
When it is in the stuck state can you please run
bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
would aid us in narrowing in on a possible cause.
Thanks
Joe
On Thu, Dec 1, 2016 at 8:44 AM, Manish G <ma...@gmail.com> wrote:
> Hi Joe,
>
> Here is what I can see in the App Log:
>
> 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
> testing.nifi.processor.hdfs.testingPutHDFS ""
> org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException:
> An error occurred while enumerating the result, check the original exception
> for details.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
> ~[hadoop-azure-2.7.2.jar:na]
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
> ~[hadoop-azure-2.7.2.jar:na]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> ~[hadoop-common-2.7.2.jar:na]
> at
> testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
> ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source) [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source) [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.8.0_101]
> at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
> Caused by: java.util.NoSuchElementException: An error occurred while
> enumerating the result, check the original exception for details.
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
> ~[azure-storage-2.0.0.jar:na]
> at
> org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
> ~[hadoop-azure-2.7.2.jar:na]
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
> ~[hadoop-azure-2.7.2.jar:na]
> ... 15 common frames omitted
> Caused by: com.microsoft.azure.storage.StorageException: Forbidden
> at
> com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
> ~[azure-storage-2.0.0.jar:na]
> ... 17 common frames omitted
> Caused by: java.lang.NullPointerException: null
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
> ~[azure-storage-2.0.0.jar:na]
> ... 18 common frames omitted
>
> Regards,
> Manish
>
> On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Manish
>>
>> Please produce and share the thread dump I mentioned.
>>
>> Thanks
>> Joe
>>
>> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I don't know why, but this is happening now more frequently. Where should
>>> I look into to find the root cause?
>>>
>>> Thanks,
>>> Manish
>>>
>>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com> wrote:
>>>>
>>>> Hi Joe,
>>>>
>>>> Thanks for the quick reply. Yes, the processor keeps running on a single
>>>> thread (even after stopping). And the number remains there even after
>>>> stopping.
>>>> Today, it happened on my customized putHDFS processor. Only thing
>>>> different in this processor is - I have added an additional attribute that
>>>> tells if the processor created the directory while loading the file on HDFS.
>>>> I don't think this should be the issue though.
>>>>
>>>> Regards,
>>>> Manish
>>>>
>>>>
>>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com> wrote:
>>>>>
>>>>> Manish
>>>>>
>>>>> When it is stuck do you see a number in the top right corner of the
>>>>> processor? When you stop it does the number remain? That number is
>>>>> telling you how many threads are still executing. Which processor are
>>>>> we talking about? When it is in the stuck state can you please run
>>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>>>>> would aid us in narrowing in on a possible cause.
>>>>>
>>>>> Thanks
>>>>> Joe
>>>>>
>>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > I have noticed that sometime a flow file gets stuck on a processor
>>>>> > for a
>>>>> > very long time for no reason and then I can not even stop the
>>>>> > processor to
>>>>> > look at the flow flow file from queue. If I click on stop, then
>>>>> > processor
>>>>> > goes into a state where I cannot start/stop the processor.
>>>>> >
>>>>> > On restarting the NiFi, the file gets processed successfully and
>>>>> > routed to
>>>>> > success queue. I checked in App log, but everything seems to be
>>>>> > normal for
>>>>> > the flow file. I don't see anything mysterious in provenance too
>>>>> > (except
>>>>> > that queue time is in hours).
>>>>> >
>>>>> > Has anyone else faced a similar issue? What else should I check to
>>>>> > identify
>>>>> > the root cause for this?
>>>>> >
>>>>> > Thanks,
>>>>> > Manish
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> With Warm Regards,
>>>> Manish
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> With Warm Regards,
>>> Manish
>
>
>
>
> --
>
>
> With Warm Regards,
> Manish
Re: Flow File Stuck for no reason
Posted by Manish G <ma...@gmail.com>.
Hi Joe,
Here is what I can see in the App Log:
2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
testing.nifi.processor.hdfs.testingPutHDFS ""
org.apache.hadoop.fs.azure.AzureException:
java.util.NoSuchElementException: An error occurred while enumerating the
result, check the original exception for details.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
~[hadoop-azure-2.7.2.jar:na]
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
~[hadoop-azure-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
~[hadoop-common-2.7.2.jar:na]
at
testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-0.7.0.jar:0.7.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
[nifi-framework-core-0.7.0.jar:0.7.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-0.7.0.jar:0.7.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-0.7.0.jar:0.7.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-0.7.0.jar:0.7.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source) [na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.8.0_101]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
Caused by: java.util.NoSuchElementException: An error occurred while
enumerating the result, check the original exception for details.
at
com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
~[azure-storage-2.0.0.jar:na]
at
org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
~[hadoop-azure-2.7.2.jar:na]
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
~[hadoop-azure-2.7.2.jar:na]
... 15 common frames omitted
Caused by: com.microsoft.azure.storage.StorageException: Forbidden
at
com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
~[azure-storage-2.0.0.jar:na]
at
com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
~[azure-storage-2.0.0.jar:na]
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
~[azure-storage-2.0.0.jar:na]
at
com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
~[azure-storage-2.0.0.jar:na]
... 17 common frames omitted
Caused by: java.lang.NullPointerException: null
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
~[azure-storage-2.0.0.jar:na]
... 18 common frames omitted
Regards,
Manish
On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <jo...@gmail.com> wrote:
> Manish
>
> Please produce and share the thread dump I mentioned.
>
> Thanks
> Joe
>
> On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
>
>> Hi,
>>
>> I don't know why, but this is happening now more frequently. Where should
>> I look into to find the root cause?
>>
>> Thanks,
>> Manish
>>
>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com> wrote:
>>
>>> Hi Joe,
>>>
>>> Thanks for the quick reply. Yes, the processor keeps running on a single
>>> thread (even after stopping). And the number remains there even after
>>> stopping.
>>> Today, it happened on my customized putHDFS processor. Only thing
>>> different in this processor is - I have added an additional attribute that
>>> tells if the processor created the directory while loading the file on
>>> HDFS. I don't think this should be the issue though.
>>>
>>> Regards,
>>> Manish
>>>
>>>
>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com> wrote:
>>>
>>>> Manish
>>>>
>>>> When it is stuck do you see a number in the top right corner of the
>>>> processor? When you stop it does the number remain? That number is
>>>> telling you how many threads are still executing. Which processor are
>>>> we talking about? When it is in the stuck state can you please run
>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>>>> would aid us in narrowing in on a possible cause.
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I have noticed that sometime a flow file gets stuck on a processor
>>>> for a
>>>> > very long time for no reason and then I can not even stop the
>>>> processor to
>>>> > look at the flow flow file from queue. If I click on stop, then
>>>> processor
>>>> > goes into a state where I cannot start/stop the processor.
>>>> >
>>>> > On restarting the NiFi, the file gets processed successfully and
>>>> routed to
>>>> > success queue. I checked in App log, but everything seems to be
>>>> normal for
>>>> > the flow file. I don't see anything mysterious in provenance too
>>>> (except
>>>> > that queue time is in hours).
>>>> >
>>>> > Has anyone else faced a similar issue? What else should I check to
>>>> identify
>>>> > the root cause for this?
>>>> >
>>>> > Thanks,
>>>> > Manish
>>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> *With Warm Regards,*
>>> *Manish*
>>>
>>
>>
>>
>> --
>>
>>
>> *With Warm Regards,*
>> *Manish*
>>
>
--
*With Warm Regards,*
*Manish*
Re: Flow File Stuck for no reason
Posted by Joe Witt <jo...@gmail.com>.
Manish
Please produce and share the thread dump I mentioned.
Thanks
Joe
On Dec 1, 2016 7:23 AM, "Manish G" <ma...@gmail.com> wrote:
> Hi,
>
> I don't know why, but this is happening now more frequently. Where should
> I look into to find the root cause?
>
> Thanks,
> Manish
>
> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com> wrote:
>
>> Hi Joe,
>>
>> Thanks for the quick reply. Yes, the processor keeps running on a single
>> thread (even after stopping). And the number remains there even after
>> stopping.
>> Today, it happened on my customized putHDFS processor. Only thing
>> different in this processor is - I have added an additional attribute that
>> tells if the processor created the directory while loading the file on
>> HDFS. I don't think this should be the issue though.
>>
>> Regards,
>> Manish
>>
>>
>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>>> Manish
>>>
>>> When it is stuck do you see a number in the top right corner of the
>>> processor? When you stop it does the number remain? That number is
>>> telling you how many threads are still executing. Which processor are
>>> we talking about? When it is in the stuck state can you please run
>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>>> would aid us in narrowing in on a possible cause.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com>
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have noticed that sometime a flow file gets stuck on a processor for
>>> a
>>> > very long time for no reason and then I can not even stop the
>>> processor to
>>> > look at the flow flow file from queue. If I click on stop, then
>>> processor
>>> > goes into a state where I cannot start/stop the processor.
>>> >
>>> > On restarting the NiFi, the file gets processed successfully and
>>> routed to
>>> > success queue. I checked in App log, but everything seems to be normal
>>> for
>>> > the flow file. I don't see anything mysterious in provenance too
>>> (except
>>> > that queue time is in hours).
>>> >
>>> > Has anyone else faced a similar issue? What else should I check to
>>> identify
>>> > the root cause for this?
>>> >
>>> > Thanks,
>>> > Manish
>>>
>>
>>
>>
>> --
>>
>>
>> *With Warm Regards,*
>> *Manish*
>>
>
>
>
> --
>
>
> *With Warm Regards,*
> *Manish*
>
Re: Flow File Stuck for no reason
Posted by Manish G <ma...@gmail.com>.
Hi,
I don't know why, but this is happening now more frequently. Where should I
look into to find the root cause?
Thanks,
Manish
On Wed, Nov 30, 2016 at 9:20 PM, Manish G <ma...@gmail.com> wrote:
> Hi Joe,
>
> Thanks for the quick reply. Yes, the processor keeps running on a single
> thread (even after stopping). And the number remains there even after
> stopping.
> Today, it happened on my customized putHDFS processor. Only thing
> different in this processor is - I have added an additional attribute that
> tells if the processor created the directory while loading the file on
> HDFS. I don't think this should be the issue though.
>
> Regards,
> Manish
>
>
> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Manish
>>
>> When it is stuck do you see a number in the top right corner of the
>> processor? When you stop it does the number remain? That number is
>> telling you how many threads are still executing. Which processor are
>> we talking about? When it is in the stuck state can you please run
>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
>> would aid us in narrowing in on a possible cause.
>>
>> Thanks
>> Joe
>>
>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I have noticed that sometime a flow file gets stuck on a processor for a
>> > very long time for no reason and then I can not even stop the processor
>> to
>> > look at the flow flow file from queue. If I click on stop, then
>> processor
>> > goes into a state where I cannot start/stop the processor.
>> >
>> > On restarting the NiFi, the file gets processed successfully and routed
>> to
>> > success queue. I checked in App log, but everything seems to be normal
>> for
>> > the flow file. I don't see anything mysterious in provenance too (except
>> > that queue time is in hours).
>> >
>> > Has anyone else faced a similar issue? What else should I check to
>> identify
>> > the root cause for this?
>> >
>> > Thanks,
>> > Manish
>>
>
>
>
> --
>
>
> *With Warm Regards,*
> *Manish*
>
--
*With Warm Regards,*
*Manish*
Re: Flow File Stuck for no reason
Posted by Manish G <ma...@gmail.com>.
Hi Joe,
Thanks for the quick reply. Yes, the processor keeps running on a single
thread (even after stopping). And the number remains there even after
stopping.
Today, it happened on my customized putHDFS processor. Only thing different
in this processor is - I have added an additional attribute that tells if
the processor created the directory while loading the file on HDFS. I don't
think this should be the issue though.
Regards,
Manish
On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <jo...@gmail.com> wrote:
> Manish
>
> When it is stuck do you see a number in the top right corner of the
> processor? When you stop it does the number remain? That number is
> telling you how many threads are still executing. Which processor are
> we talking about? When it is in the stuck state can you please run
> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
> would aid us in narrowing in on a possible cause.
>
> Thanks
> Joe
>
> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have noticed that sometime a flow file gets stuck on a processor for a
> > very long time for no reason and then I can not even stop the processor
> to
> > look at the flow flow file from queue. If I click on stop, then processor
> > goes into a state where I cannot start/stop the processor.
> >
> > On restarting the NiFi, the file gets processed successfully and routed
> to
> > success queue. I checked in App log, but everything seems to be normal
> for
> > the flow file. I don't see anything mysterious in provenance too (except
> > that queue time is in hours).
> >
> > Has anyone else faced a similar issue? What else should I check to
> identify
> > the root cause for this?
> >
> > Thanks,
> > Manish
>
--
*With Warm Regards,*
*Manish*
Re: Flow File Stuck for no reason
Posted by Joe Witt <jo...@gmail.com>.
Manish
When it is stuck do you see a number in the top right corner of the
processor? When you stop it does the number remain? That number is
telling you how many threads are still executing. Which processor are
we talking about? When it is in the stuck state can you please run
bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that
would aid us in narrowing in on a possible cause.
Thanks
Joe
On Wed, Nov 30, 2016 at 7:02 PM, Manish G <ma...@gmail.com> wrote:
>
> Hi,
>
> I have noticed that sometime a flow file gets stuck on a processor for a
> very long time for no reason and then I can not even stop the processor to
> look at the flow flow file from queue. If I click on stop, then processor
> goes into a state where I cannot start/stop the processor.
>
> On restarting the NiFi, the file gets processed successfully and routed to
> success queue. I checked in App log, but everything seems to be normal for
> the flow file. I don't see anything mysterious in provenance too (except
> that queue time is in hours).
>
> Has anyone else faced a similar issue? What else should I check to identify
> the root cause for this?
>
> Thanks,
> Manish