You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Suman B N <su...@gmail.com> on 2019/05/09 09:30:11 UTC
Apache Nifi issues
Team,
We are running a 3 node nifi cluster in docker. Version is 1.8.
Everything has been running smoothly from the last 2-3 months. But from
last 2 days, we are observing a couple of issues and those are explained
below:
- *Load Balanced Client Thread throwing exceptions continuously.* Nifi
logs are filled with below errors and making the disk full very soon.
Stacktrace:
2019-05-09 09:06:26,166 ERROR [Load-Balanced Client Thread-1]
o.a.n.c.queue.SwappablePriorityQueue Updated Size of Queue Unacknowledged
from FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes], Swap Queue=[0, 0
Bytes], Swap Files=[0], Unacknowledged=[-484, -50212519 Bytes] ] to
FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes], Swap Queue=[0, 0 Bytes],
Swap Files=[0], Unacknowledged=[-484, -50212519 Bytes] ]
java.lang.RuntimeException: Cannot create negative queue size
at
org.apache.nifi.controller.queue.SwappablePriorityQueue.logIfNegative(SwappablePriorityQueue.java:925)
at
org.apache.nifi.controller.queue.SwappablePriorityQueue.incrementUnacknowledgedQueueSize(SwappablePriorityQueue.java:915)
at
org.apache.nifi.controller.queue.SwappablePriorityQueue.acknowledge(SwappablePriorityQueue.java:417)
at
org.apache.nifi.controller.queue.clustered.partition.RemoteQueuePartition$2.onTransactionComplete(RemoteQueuePartition.java:210)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:259)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:76)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
- *Flow files are stuck in queue.* Probably its this issue - NIFI-5919
<https://issues.apache.org/jira/browse/NIFI-5919> which is fixed in 1.9.
The flow files are stuck in the queue(Load balance by attribute) and are
not read by the next downstream processor(MergeRecord with CSVReader and
CSVRecordSetWriter). From the Nifi UI, it appears that flow files are in
the queue but when tried to list queue it says "Queue has no flow files".
Attempting to empty queue also gives the exact message. I have tried below
action items but all in vain:
- Restarting the downstream and upstream(ConvertRecord) processor.
- Disabled and enabled CSVReader and CSVRecordSetWriter.
- Disabled load balancing.
How can I debug this or resolve this? I am pretty new to nifi. Trying my
best to understand the flow file life cycle and internal architecture of
nifi.
Any leads or assistance to solve the above two issues is very much
appreciated.
--
*Suman*
*Tathastu*
Re: Apache Nifi issues
Posted by Joe Witt <jo...@gmail.com>.
Suman
The community resolved these (and many other) issues in Apache NiFi 1.9.x
line.
There are vendor offerings that have addressed it similarly but which may
be more direct to the line you're talking about.
Thanks
On Thu, May 9, 2019 at 11:43 AM Suman B N <su...@gmail.com> wrote:
> Any turn around solutions in version 1.8 to resolve these? If not then I
> will plan to upgrade soon. If there are any quicker solutions which can be
> applied in 1.8 without any data loss then I ll upgrade it later.
>
> On Thursday, May 9, 2019, Joe Witt <jo...@gmail.com> wrote:
>
>> Suman
>>
>> Yeah it looks related to the queue/load balance fixes. Latest release
>> should be much better for you.
>>
>> Thanks
>>
>>
>> On Thu, May 9, 2019, 5:30 AM Suman B N <su...@gmail.com> wrote:
>>
>>> Team,
>>> We are running a 3 node nifi cluster in docker. Version is 1.8.
>>> Everything has been running smoothly from the last 2-3 months. But from
>>> last 2 days, we are observing a couple of issues and those are explained
>>> below:
>>>
>>>
>>> - *Load Balanced Client Thread throwing exceptions continuously.*
>>> Nifi logs are filled with below errors and making the disk full very soon.
>>>
>>> Stacktrace:
>>>
>>> 2019-05-09 09:06:26,166 ERROR [Load-Balanced Client Thread-1]
>>> o.a.n.c.queue.SwappablePriorityQueue Updated Size of Queue Unacknowledged
>>> from FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes], Swap Queue=[0, 0
>>> Bytes], Swap Files=[0], Unacknowledged=[-484, -50212519 Bytes] ] to
>>> FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes], Swap Queue=[0, 0 Bytes],
>>> Swap Files=[0], Unacknowledged=[-484, -50212519 Bytes] ]
>>>
>>> java.lang.RuntimeException: Cannot create negative queue size
>>>
>>> at
>>> org.apache.nifi.controller.queue.SwappablePriorityQueue.logIfNegative(SwappablePriorityQueue.java:925)
>>>
>>> at
>>> org.apache.nifi.controller.queue.SwappablePriorityQueue.incrementUnacknowledgedQueueSize(SwappablePriorityQueue.java:915)
>>>
>>> at
>>> org.apache.nifi.controller.queue.SwappablePriorityQueue.acknowledge(SwappablePriorityQueue.java:417)
>>>
>>> at
>>> org.apache.nifi.controller.queue.clustered.partition.RemoteQueuePartition$2.onTransactionComplete(RemoteQueuePartition.java:210)
>>>
>>> at
>>> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:259)
>>>
>>> at
>>> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:76)
>>>
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>> - *Flow files are stuck in queue.* Probably its this issue -
>>> NIFI-5919 <https://issues.apache.org/jira/browse/NIFI-5919> which is
>>> fixed in 1.9.
>>>
>>> The flow files are stuck in the queue(Load balance by attribute) and are
>>> not read by the next downstream processor(MergeRecord with CSVReader and
>>> CSVRecordSetWriter). From the Nifi UI, it appears that flow files are in
>>> the queue but when tried to list queue it says "Queue has no flow files".
>>> Attempting to empty queue also gives the exact message. I have tried below
>>> action items but all in vain:
>>>
>>> - Restarting the downstream and upstream(ConvertRecord) processor.
>>> - Disabled and enabled CSVReader and CSVRecordSetWriter.
>>> - Disabled load balancing.
>>>
>>> How can I debug this or resolve this? I am pretty new to nifi. Trying my
>>> best to understand the flow file life cycle and internal architecture of
>>> nifi.
>>>
>>> Any leads or assistance to solve the above two issues is very much
>>> appreciated.
>>> --
>>> *Suman*
>>> *Tathastu*
>>>
>>
>
> --
> *Suman*
> *OlaCabs*
>
>
Apache Nifi issues
Posted by Suman B N <su...@gmail.com>.
Any turn around solutions in version 1.8 to resolve these? If not then I
will plan to upgrade soon. If there are any quicker solutions which can be
applied in 1.8 without any data loss then I ll upgrade it later.
On Thursday, May 9, 2019, Joe Witt <jo...@gmail.com> wrote:
> Suman
>
> Yeah it looks related to the queue/load balance fixes. Latest release
> should be much better for you.
>
> Thanks
>
>
> On Thu, May 9, 2019, 5:30 AM Suman B N <su...@gmail.com> wrote:
>
>> Team,
>> We are running a 3 node nifi cluster in docker. Version is 1.8.
>> Everything has been running smoothly from the last 2-3 months. But from
>> last 2 days, we are observing a couple of issues and those are explained
>> below:
>>
>>
>> - *Load Balanced Client Thread throwing exceptions continuously.*
>> Nifi logs are filled with below errors and making the disk full very soon.
>>
>> Stacktrace:
>>
>> 2019-05-09 09:06:26,166 ERROR [Load-Balanced Client Thread-1]
>> o.a.n.c.queue.SwappablePriorityQueue Updated Size of Queue
>> Unacknowledged from FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes], Swap
>> Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-484, -50212519
>> Bytes] ] to FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes], Swap
>> Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-484, -50212519 Bytes] ]
>>
>> java.lang.RuntimeException: Cannot create negative queue size
>>
>> at org.apache.nifi.controller.queue.SwappablePriorityQueue.logI
>> fNegative(SwappablePriorityQueue.java:925)
>>
>> at org.apache.nifi.controller.queue.SwappablePriorityQueue.incr
>> ementUnacknowledgedQueueSize(SwappablePriorityQueue.java:915)
>>
>> at org.apache.nifi.controller.queue.SwappablePriorityQueue.ackn
>> owledge(SwappablePriorityQueue.java:417)
>>
>> at org.apache.nifi.controller.queue.clustered.partition.RemoteQ
>> ueuePartition$2.onTransactionComplete(RemoteQueuePartition.java:210)
>>
>> at org.apache.nifi.controller.queue.clustered.client.async.nio.
>> NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:259)
>>
>> at org.apache.nifi.controller.queue.clustered.client.async.nio.
>> NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:76)
>>
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executor
>> s.java:511)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1149)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:624)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>> - *Flow files are stuck in queue.* Probably its this issue - NIFI-5919
>> <https://issues.apache.org/jira/browse/NIFI-5919> which is fixed in
>> 1.9.
>>
>> The flow files are stuck in the queue(Load balance by attribute) and are
>> not read by the next downstream processor(MergeRecord with CSVReader and
>> CSVRecordSetWriter). From the Nifi UI, it appears that flow files are in
>> the queue but when tried to list queue it says "Queue has no flow files".
>> Attempting to empty queue also gives the exact message. I have tried below
>> action items but all in vain:
>>
>> - Restarting the downstream and upstream(ConvertRecord) processor.
>> - Disabled and enabled CSVReader and CSVRecordSetWriter.
>> - Disabled load balancing.
>>
>> How can I debug this or resolve this? I am pretty new to nifi. Trying my
>> best to understand the flow file life cycle and internal architecture of
>> nifi.
>>
>> Any leads or assistance to solve the above two issues is very much
>> appreciated.
>> --
>> *Suman*
>> *Tathastu*
>>
>
--
*Suman*
*OlaCabs*
Re: Apache Nifi issues
Posted by Joe Witt <jo...@gmail.com>.
Suman
Yeah it looks related to the queue/load balance fixes. Latest release
should be much better for you.
Thanks
On Thu, May 9, 2019, 5:30 AM Suman B N <su...@gmail.com> wrote:
> Team,
> We are running a 3 node nifi cluster in docker. Version is 1.8.
> Everything has been running smoothly from the last 2-3 months. But from
> last 2 days, we are observing a couple of issues and those are explained
> below:
>
>
> - *Load Balanced Client Thread throwing exceptions continuously.* Nifi
> logs are filled with below errors and making the disk full very soon.
>
> Stacktrace:
>
> 2019-05-09 09:06:26,166 ERROR [Load-Balanced Client Thread-1]
> o.a.n.c.queue.SwappablePriorityQueue Updated Size of Queue Unacknowledged
> from FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes], Swap Queue=[0, 0
> Bytes], Swap Files=[0], Unacknowledged=[-484, -50212519 Bytes] ] to
> FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes], Swap Queue=[0, 0 Bytes],
> Swap Files=[0], Unacknowledged=[-484, -50212519 Bytes] ]
>
> java.lang.RuntimeException: Cannot create negative queue size
>
> at
> org.apache.nifi.controller.queue.SwappablePriorityQueue.logIfNegative(SwappablePriorityQueue.java:925)
>
> at
> org.apache.nifi.controller.queue.SwappablePriorityQueue.incrementUnacknowledgedQueueSize(SwappablePriorityQueue.java:915)
>
> at
> org.apache.nifi.controller.queue.SwappablePriorityQueue.acknowledge(SwappablePriorityQueue.java:417)
>
> at
> org.apache.nifi.controller.queue.clustered.partition.RemoteQueuePartition$2.onTransactionComplete(RemoteQueuePartition.java:210)
>
> at
> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:259)
>
> at
> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:76)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> - *Flow files are stuck in queue.* Probably its this issue - NIFI-5919
> <https://issues.apache.org/jira/browse/NIFI-5919> which is fixed in
> 1.9.
>
> The flow files are stuck in the queue(Load balance by attribute) and are
> not read by the next downstream processor(MergeRecord with CSVReader and
> CSVRecordSetWriter). From the Nifi UI, it appears that flow files are in
> the queue but when tried to list queue it says "Queue has no flow files".
> Attempting to empty queue also gives the exact message. I have tried below
> action items but all in vain:
>
> - Restarting the downstream and upstream(ConvertRecord) processor.
> - Disabled and enabled CSVReader and CSVRecordSetWriter.
> - Disabled load balancing.
>
> How can I debug this or resolve this? I am pretty new to nifi. Trying my
> best to understand the flow file life cycle and internal architecture of
> nifi.
>
> Any leads or assistance to solve the above two issues is very much
> appreciated.
> --
> *Suman*
> *Tathastu*
>