You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by scott <tc...@gmail.com> on 2021/07/31 23:00:14 UTC
odd performance behavior 1.14
Hi All,
I upgraded to 1.14 last week and within a few days I started to see some
pretty odd behavior, I'm hoping someone has either seen it before or could
point me to a deeper level of troubleshooting.
Here are the symptoms I observed.
* Performance issues:
Processor performance very poor. Even simple processors like router and
updateattribute went from being able to process 100,000recs/min to
100recs/min or stop all together, but not consistently.
Processors needing to be force killed, even simple ones like
updateattribute.
* Weirdness. One of my routers lost its mind and didn't recognize the
routes configured anymore. It changed all the arrows to dotted lines except
for the default. I ended up copying it and replacing it with the copy, no
changes mind you, but it worked fine.
* Errors: I have not found any obvious errors in the nifi logs that could
explain this, but one error keeps repeating in the logs: SQLServerDriver is
not found . I have dozens of processors that use SQL Server, all seem to be
working fine. This is not tied to a particular processor's configuration. I
don't think this is related.
* Server resources fine. I use htop and sar to troubleshoot hardware issues
usually, all looks normal. I added 1/3 more memory to the JVM, now at 24GB,
just for good measure, but that had no effect.
Is it possible there are some hidden performance issues going on within the
JVM I need a special tool to see?
Any help would be greatly appreciated.
Thanks,
Scott
Re: odd performance behavior 1.14
Posted by Etienne Jouvin <la...@gmail.com>.
Hello.
I may have notice something similar.
And it was a matter of disk Space for the provenance repository.
There was not enough space, and flowfile with large content....
NiFi was waiting to free some spaces.
Regards
Etienne Jouvin
Le lun. 2 août 2021 à 20:20, scott <tc...@gmail.com> a écrit :
> I'm using openjdk-11.0.7.10-4 as I was on the previous version of NiFi.
> I'll look around for a free Java profiler to use to dig deeper.
> Thanks,
> Scott
>
> On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <jo...@gmail.com> wrote:
>
>> Scott
>>
>> Nope this sounds pretty dang unique
>>
>> What JVM? May need to attach a profiler.
>>
>> I have seen buried exceptions happening at massive rates causing horrid
>> performance among a few other scenarios but nothing specific to 1.14
>>
>> thanks
>>
>> On Sat, Jul 31, 2021 at 4:01 PM scott <tc...@gmail.com> wrote:
>>
>>> Hi All,
>>> I upgraded to 1.14 last week and within a few days I started to see some
>>> pretty odd behavior, I'm hoping someone has either seen it before or could
>>> point me to a deeper level of troubleshooting.
>>>
>>> Here are the symptoms I observed.
>>> * Performance issues:
>>> Processor performance very poor. Even simple processors like router
>>> and updateattribute went from being able to process 100,000recs/min to
>>> 100recs/min or stop all together, but not consistently.
>>> Processors needing to be force killed, even simple ones like
>>> updateattribute.
>>>
>>> * Weirdness. One of my routers lost its mind and didn't recognize the
>>> routes configured anymore. It changed all the arrows to dotted lines except
>>> for the default. I ended up copying it and replacing it with the copy, no
>>> changes mind you, but it worked fine.
>>>
>>> * Errors: I have not found any obvious errors in the nifi logs that
>>> could explain this, but one error keeps repeating in the
>>> logs: SQLServerDriver is not found . I have dozens of processors that use
>>> SQL Server, all seem to be working fine. This is not tied to a particular
>>> processor's configuration. I don't think this is related.
>>>
>>> * Server resources fine. I use htop and sar to troubleshoot hardware
>>> issues usually, all looks normal. I added 1/3 more memory to the JVM, now
>>> at 24GB, just for good measure, but that had no effect.
>>>
>>> Is it possible there are some hidden performance issues going on within
>>> the JVM I need a special tool to see?
>>>
>>> Any help would be greatly appreciated.
>>>
>>> Thanks,
>>> Scott
>>>
>>>
>>>
>>>
Re: odd performance behavior 1.14
Posted by Russell Bateman <ru...@windofkeltia.com>.
(I'm sorry, I meant to say that I read that the Java Flight Recorder is
now available for unlicensed--in the original Oracle sense--for use.)
On 8/2/21 12:44 PM, Russell Bateman wrote:
> Scott,
>
> I believe I read somewhere in the last year. I found this:
>
> https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u
>
> And, I used it to look into a problem I had with Apache NiFi a few
> years ago. For what it's worth, my experience is recorded here:
>
> https://www.javahotchocolate.com/notes/jfr.html
>
> Hope some of this helps.
>
> Russ
>
> On 8/2/21 12:20 PM, scott wrote:
>> I'm using openjdk-11.0.7.10-4 as I was on the previous version of
>> NiFi. I'll look around for a free Java profiler to use to dig deeper.
>> Thanks,
>> Scott
>>
>> On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <joe.witt@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Scott
>>
>> Nope this sounds pretty dang unique
>>
>> What JVM? May need to attach a profiler.
>>
>> I have seen buried exceptions happening at massive rates causing
>> horrid performance among a few other scenarios but nothing
>> specific to 1.14
>>
>> thanks
>>
>> On Sat, Jul 31, 2021 at 4:01 PM scott <tcots8888@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hi All,
>> I upgraded to 1.14 last week and within a few days I started
>> to see some pretty odd behavior, I'm hoping someone has
>> either seen it before or could point me to a deeper level of
>> troubleshooting.
>>
>> Here are the symptoms I observed.
>> * Performance issues:
>> Processor performance very poor. Even simple processors
>> like router and updateattribute went from being able to
>> process 100,000recs/min to 100recs/min or stop all together,
>> but not consistently.
>> Processors needing to be force killed, even simple ones
>> like updateattribute.
>>
>> * Weirdness. One of my routers lost its mind and
>> didn't recognize the routes configured anymore. It changed
>> all the arrows to dotted lines except for the default. I
>> ended up copying it and replacing it with the copy, no
>> changes mind you, but it worked fine.
>>
>> * Errors: I have not found any obvious errors in the nifi
>> logs that could explain this, but one error keeps repeating
>> in the logs: SQLServerDriver is not found . I have dozens of
>> processors that use SQL Server, all seem to be working fine.
>> This is not tied to a particular processor's configuration. I
>> don't think this is related.
>>
>> * Server resources fine. I use htop and sar to troubleshoot
>> hardware issues usually, all looks normal. I added 1/3 more
>> memory to the JVM, now at 24GB, just for good measure, but
>> that had no effect.
>>
>> Is it possible there are some hidden performance issues going
>> on within the JVM I need a special tool to see?
>>
>> Any help would be greatly appreciated.
>>
>> Thanks,
>> Scott
>>
>>
>
Re: odd performance behavior 1.14
Posted by Russell Bateman <ru...@windofkeltia.com>.
Scott,
I believe I read somewhere in the last year. I found this:
https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u
And, I used it to look into a problem I had with Apache NiFi a few years
ago. For what it's worth, my experience is recorded here:
https://www.javahotchocolate.com/notes/jfr.html
Hope some of this helps.
Russ
On 8/2/21 12:20 PM, scott wrote:
> I'm using openjdk-11.0.7.10-4 as I was on the previous version of
> NiFi. I'll look around for a free Java profiler to use to dig deeper.
> Thanks,
> Scott
>
> On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <joe.witt@gmail.com
> <ma...@gmail.com>> wrote:
>
> Scott
>
> Nope this sounds pretty dang unique
>
> What JVM? May need to attach a profiler.
>
> I have seen buried exceptions happening at massive rates causing
> horrid performance among a few other scenarios but nothing
> specific to 1.14
>
> thanks
>
> On Sat, Jul 31, 2021 at 4:01 PM scott <tcots8888@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi All,
> I upgraded to 1.14 last week and within a few days I started
> to see some pretty odd behavior, I'm hoping someone has either
> seen it before or could point me to a deeper level of
> troubleshooting.
>
> Here are the symptoms I observed.
> * Performance issues:
> Processor performance very poor. Even simple processors
> like router and updateattribute went from being able to
> process 100,000recs/min to 100recs/min or stop all together,
> but not consistently.
> Processors needing to be force killed, even simple ones
> like updateattribute.
>
> * Weirdness. One of my routers lost its mind and
> didn't recognize the routes configured anymore. It changed all
> the arrows to dotted lines except for the default. I ended up
> copying it and replacing it with the copy, no changes
> mind you, but it worked fine.
>
> * Errors: I have not found any obvious errors in the nifi logs
> that could explain this, but one error keeps repeating in the
> logs: SQLServerDriver is not found . I have dozens of
> processors that use SQL Server, all seem to be working fine.
> This is not tied to a particular processor's configuration. I
> don't think this is related.
>
> * Server resources fine. I use htop and sar to troubleshoot
> hardware issues usually, all looks normal. I added 1/3 more
> memory to the JVM, now at 24GB, just for good measure, but
> that had no effect.
>
> Is it possible there are some hidden performance issues going
> on within the JVM I need a special tool to see?
>
> Any help would be greatly appreciated.
>
> Thanks,
> Scott
>
>
Re: odd performance behavior 1.14
Posted by scott <tc...@gmail.com>.
I'm using openjdk-11.0.7.10-4 as I was on the previous version of NiFi.
I'll look around for a free Java profiler to use to dig deeper.
Thanks,
Scott
On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <jo...@gmail.com> wrote:
> Scott
>
> Nope this sounds pretty dang unique
>
> What JVM? May need to attach a profiler.
>
> I have seen buried exceptions happening at massive rates causing horrid
> performance among a few other scenarios but nothing specific to 1.14
>
> thanks
>
> On Sat, Jul 31, 2021 at 4:01 PM scott <tc...@gmail.com> wrote:
>
>> Hi All,
>> I upgraded to 1.14 last week and within a few days I started to see some
>> pretty odd behavior, I'm hoping someone has either seen it before or could
>> point me to a deeper level of troubleshooting.
>>
>> Here are the symptoms I observed.
>> * Performance issues:
>> Processor performance very poor. Even simple processors like router
>> and updateattribute went from being able to process 100,000recs/min to
>> 100recs/min or stop all together, but not consistently.
>> Processors needing to be force killed, even simple ones like
>> updateattribute.
>>
>> * Weirdness. One of my routers lost its mind and didn't recognize the
>> routes configured anymore. It changed all the arrows to dotted lines except
>> for the default. I ended up copying it and replacing it with the copy, no
>> changes mind you, but it worked fine.
>>
>> * Errors: I have not found any obvious errors in the nifi logs that could
>> explain this, but one error keeps repeating in the logs: SQLServerDriver is
>> not found . I have dozens of processors that use SQL Server, all seem to be
>> working fine. This is not tied to a particular processor's configuration. I
>> don't think this is related.
>>
>> * Server resources fine. I use htop and sar to troubleshoot hardware
>> issues usually, all looks normal. I added 1/3 more memory to the JVM, now
>> at 24GB, just for good measure, but that had no effect.
>>
>> Is it possible there are some hidden performance issues going on within
>> the JVM I need a special tool to see?
>>
>> Any help would be greatly appreciated.
>>
>> Thanks,
>> Scott
>>
>>
>>
>>
Re: odd performance behavior 1.14
Posted by Joe Witt <jo...@gmail.com>.
Scott
Nope this sounds pretty dang unique
What JVM? May need to attach a profiler.
I have seen buried exceptions happening at massive rates causing horrid
performance among a few other scenarios but nothing specific to 1.14
thanks
On Sat, Jul 31, 2021 at 4:01 PM scott <tc...@gmail.com> wrote:
> Hi All,
> I upgraded to 1.14 last week and within a few days I started to see some
> pretty odd behavior, I'm hoping someone has either seen it before or could
> point me to a deeper level of troubleshooting.
>
> Here are the symptoms I observed.
> * Performance issues:
> Processor performance very poor. Even simple processors like router
> and updateattribute went from being able to process 100,000recs/min to
> 100recs/min or stop all together, but not consistently.
> Processors needing to be force killed, even simple ones like
> updateattribute.
>
> * Weirdness. One of my routers lost its mind and didn't recognize the
> routes configured anymore. It changed all the arrows to dotted lines except
> for the default. I ended up copying it and replacing it with the copy, no
> changes mind you, but it worked fine.
>
> * Errors: I have not found any obvious errors in the nifi logs that could
> explain this, but one error keeps repeating in the logs: SQLServerDriver is
> not found . I have dozens of processors that use SQL Server, all seem to be
> working fine. This is not tied to a particular processor's configuration. I
> don't think this is related.
>
> * Server resources fine. I use htop and sar to troubleshoot hardware
> issues usually, all looks normal. I added 1/3 more memory to the JVM, now
> at 24GB, just for good measure, but that had no effect.
>
> Is it possible there are some hidden performance issues going on within
> the JVM I need a special tool to see?
>
> Any help would be greatly appreciated.
>
> Thanks,
> Scott
>
>
>
>