You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Joe Witt <jo...@gmail.com> on 2021/08/01 02:56:10 UTC
Re: odd performance behavior 1.14
Scott
Nope this sounds pretty dang unique
What JVM? May need to attach a profiler.
I have seen buried exceptions happening at massive rates causing horrid
performance among a few other scenarios but nothing specific to 1.14
thanks
On Sat, Jul 31, 2021 at 4:01 PM scott <tc...@gmail.com> wrote:
> Hi All,
> I upgraded to 1.14 last week and within a few days I started to see some
> pretty odd behavior, I'm hoping someone has either seen it before or could
> point me to a deeper level of troubleshooting.
>
> Here are the symptoms I observed.
> * Performance issues:
> Processor performance very poor. Even simple processors like router
> and updateattribute went from being able to process 100,000recs/min to
> 100recs/min or stop all together, but not consistently.
> Processors needing to be force killed, even simple ones like
> updateattribute.
>
> * Weirdness. One of my routers lost its mind and didn't recognize the
> routes configured anymore. It changed all the arrows to dotted lines except
> for the default. I ended up copying it and replacing it with the copy, no
> changes mind you, but it worked fine.
>
> * Errors: I have not found any obvious errors in the nifi logs that could
> explain this, but one error keeps repeating in the logs: SQLServerDriver is
> not found . I have dozens of processors that use SQL Server, all seem to be
> working fine. This is not tied to a particular processor's configuration. I
> don't think this is related.
>
> * Server resources fine. I use htop and sar to troubleshoot hardware
> issues usually, all looks normal. I added 1/3 more memory to the JVM, now
> at 24GB, just for good measure, but that had no effect.
>
> Is it possible there are some hidden performance issues going on within
> the JVM I need a special tool to see?
>
> Any help would be greatly appreciated.
>
> Thanks,
> Scott
>
>
>
>
Re: odd performance behavior 1.14
Posted by Etienne Jouvin <la...@gmail.com>.
Hello.
I may have notice something similar.
And it was a matter of disk Space for the provenance repository.
There was not enough space, and flowfile with large content....
NiFi was waiting to free some spaces.
Regards
Etienne Jouvin
Le lun. 2 août 2021 à 20:20, scott <tc...@gmail.com> a écrit :
> I'm using openjdk-11.0.7.10-4 as I was on the previous version of NiFi.
> I'll look around for a free Java profiler to use to dig deeper.
> Thanks,
> Scott
>
> On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <jo...@gmail.com> wrote:
>
>> Scott
>>
>> Nope this sounds pretty dang unique
>>
>> What JVM? May need to attach a profiler.
>>
>> I have seen buried exceptions happening at massive rates causing horrid
>> performance among a few other scenarios but nothing specific to 1.14
>>
>> thanks
>>
>> On Sat, Jul 31, 2021 at 4:01 PM scott <tc...@gmail.com> wrote:
>>
>>> Hi All,
>>> I upgraded to 1.14 last week and within a few days I started to see some
>>> pretty odd behavior, I'm hoping someone has either seen it before or could
>>> point me to a deeper level of troubleshooting.
>>>
>>> Here are the symptoms I observed.
>>> * Performance issues:
>>> Processor performance very poor. Even simple processors like router
>>> and updateattribute went from being able to process 100,000recs/min to
>>> 100recs/min or stop all together, but not consistently.
>>> Processors needing to be force killed, even simple ones like
>>> updateattribute.
>>>
>>> * Weirdness. One of my routers lost its mind and didn't recognize the
>>> routes configured anymore. It changed all the arrows to dotted lines except
>>> for the default. I ended up copying it and replacing it with the copy, no
>>> changes mind you, but it worked fine.
>>>
>>> * Errors: I have not found any obvious errors in the nifi logs that
>>> could explain this, but one error keeps repeating in the
>>> logs: SQLServerDriver is not found . I have dozens of processors that use
>>> SQL Server, all seem to be working fine. This is not tied to a particular
>>> processor's configuration. I don't think this is related.
>>>
>>> * Server resources fine. I use htop and sar to troubleshoot hardware
>>> issues usually, all looks normal. I added 1/3 more memory to the JVM, now
>>> at 24GB, just for good measure, but that had no effect.
>>>
>>> Is it possible there are some hidden performance issues going on within
>>> the JVM I need a special tool to see?
>>>
>>> Any help would be greatly appreciated.
>>>
>>> Thanks,
>>> Scott
>>>
>>>
>>>
>>>
Re: odd performance behavior 1.14
Posted by Russell Bateman <ru...@windofkeltia.com>.
(I'm sorry, I meant to say that I read that the Java Flight Recorder is
now available for unlicensed--in the original Oracle sense--for use.)
On 8/2/21 12:44 PM, Russell Bateman wrote:
> Scott,
>
> I believe I read somewhere in the last year. I found this:
>
> https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u
>
> And, I used it to look into a problem I had with Apache NiFi a few
> years ago. For what it's worth, my experience is recorded here:
>
> https://www.javahotchocolate.com/notes/jfr.html
>
> Hope some of this helps.
>
> Russ
>
> On 8/2/21 12:20 PM, scott wrote:
>> I'm using openjdk-11.0.7.10-4 as I was on the previous version of
>> NiFi. I'll look around for a free Java profiler to use to dig deeper.
>> Thanks,
>> Scott
>>
>> On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <joe.witt@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Scott
>>
>> Nope this sounds pretty dang unique
>>
>> What JVM? May need to attach a profiler.
>>
>> I have seen buried exceptions happening at massive rates causing
>> horrid performance among a few other scenarios but nothing
>> specific to 1.14
>>
>> thanks
>>
>> On Sat, Jul 31, 2021 at 4:01 PM scott <tcots8888@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hi All,
>> I upgraded to 1.14 last week and within a few days I started
>> to see some pretty odd behavior, I'm hoping someone has
>> either seen it before or could point me to a deeper level of
>> troubleshooting.
>>
>> Here are the symptoms I observed.
>> * Performance issues:
>> Processor performance very poor. Even simple processors
>> like router and updateattribute went from being able to
>> process 100,000recs/min to 100recs/min or stop all together,
>> but not consistently.
>> Processors needing to be force killed, even simple ones
>> like updateattribute.
>>
>> * Weirdness. One of my routers lost its mind and
>> didn't recognize the routes configured anymore. It changed
>> all the arrows to dotted lines except for the default. I
>> ended up copying it and replacing it with the copy, no
>> changes mind you, but it worked fine.
>>
>> * Errors: I have not found any obvious errors in the nifi
>> logs that could explain this, but one error keeps repeating
>> in the logs: SQLServerDriver is not found . I have dozens of
>> processors that use SQL Server, all seem to be working fine.
>> This is not tied to a particular processor's configuration. I
>> don't think this is related.
>>
>> * Server resources fine. I use htop and sar to troubleshoot
>> hardware issues usually, all looks normal. I added 1/3 more
>> memory to the JVM, now at 24GB, just for good measure, but
>> that had no effect.
>>
>> Is it possible there are some hidden performance issues going
>> on within the JVM I need a special tool to see?
>>
>> Any help would be greatly appreciated.
>>
>> Thanks,
>> Scott
>>
>>
>
Re: odd performance behavior 1.14
Posted by Russell Bateman <ru...@windofkeltia.com>.
Scott,
I believe I read somewhere in the last year. I found this:
https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u
And, I used it to look into a problem I had with Apache NiFi a few years
ago. For what it's worth, my experience is recorded here:
https://www.javahotchocolate.com/notes/jfr.html
Hope some of this helps.
Russ
On 8/2/21 12:20 PM, scott wrote:
> I'm using openjdk-11.0.7.10-4 as I was on the previous version of
> NiFi. I'll look around for a free Java profiler to use to dig deeper.
> Thanks,
> Scott
>
> On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <joe.witt@gmail.com
> <ma...@gmail.com>> wrote:
>
> Scott
>
> Nope this sounds pretty dang unique
>
> What JVM? May need to attach a profiler.
>
> I have seen buried exceptions happening at massive rates causing
> horrid performance among a few other scenarios but nothing
> specific to 1.14
>
> thanks
>
> On Sat, Jul 31, 2021 at 4:01 PM scott <tcots8888@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi All,
> I upgraded to 1.14 last week and within a few days I started
> to see some pretty odd behavior, I'm hoping someone has either
> seen it before or could point me to a deeper level of
> troubleshooting.
>
> Here are the symptoms I observed.
> * Performance issues:
> Processor performance very poor. Even simple processors
> like router and updateattribute went from being able to
> process 100,000recs/min to 100recs/min or stop all together,
> but not consistently.
> Processors needing to be force killed, even simple ones
> like updateattribute.
>
> * Weirdness. One of my routers lost its mind and
> didn't recognize the routes configured anymore. It changed all
> the arrows to dotted lines except for the default. I ended up
> copying it and replacing it with the copy, no changes
> mind you, but it worked fine.
>
> * Errors: I have not found any obvious errors in the nifi logs
> that could explain this, but one error keeps repeating in the
> logs: SQLServerDriver is not found . I have dozens of
> processors that use SQL Server, all seem to be working fine.
> This is not tied to a particular processor's configuration. I
> don't think this is related.
>
> * Server resources fine. I use htop and sar to troubleshoot
> hardware issues usually, all looks normal. I added 1/3 more
> memory to the JVM, now at 24GB, just for good measure, but
> that had no effect.
>
> Is it possible there are some hidden performance issues going
> on within the JVM I need a special tool to see?
>
> Any help would be greatly appreciated.
>
> Thanks,
> Scott
>
>
Re: odd performance behavior 1.14
Posted by scott <tc...@gmail.com>.
I'm using openjdk-11.0.7.10-4 as I was on the previous version of NiFi.
I'll look around for a free Java profiler to use to dig deeper.
Thanks,
Scott
On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <jo...@gmail.com> wrote:
> Scott
>
> Nope this sounds pretty dang unique
>
> What JVM? May need to attach a profiler.
>
> I have seen buried exceptions happening at massive rates causing horrid
> performance among a few other scenarios but nothing specific to 1.14
>
> thanks
>
> On Sat, Jul 31, 2021 at 4:01 PM scott <tc...@gmail.com> wrote:
>
>> Hi All,
>> I upgraded to 1.14 last week and within a few days I started to see some
>> pretty odd behavior, I'm hoping someone has either seen it before or could
>> point me to a deeper level of troubleshooting.
>>
>> Here are the symptoms I observed.
>> * Performance issues:
>> Processor performance very poor. Even simple processors like router
>> and updateattribute went from being able to process 100,000recs/min to
>> 100recs/min or stop all together, but not consistently.
>> Processors needing to be force killed, even simple ones like
>> updateattribute.
>>
>> * Weirdness. One of my routers lost its mind and didn't recognize the
>> routes configured anymore. It changed all the arrows to dotted lines except
>> for the default. I ended up copying it and replacing it with the copy, no
>> changes mind you, but it worked fine.
>>
>> * Errors: I have not found any obvious errors in the nifi logs that could
>> explain this, but one error keeps repeating in the logs: SQLServerDriver is
>> not found . I have dozens of processors that use SQL Server, all seem to be
>> working fine. This is not tied to a particular processor's configuration. I
>> don't think this is related.
>>
>> * Server resources fine. I use htop and sar to troubleshoot hardware
>> issues usually, all looks normal. I added 1/3 more memory to the JVM, now
>> at 24GB, just for good measure, but that had no effect.
>>
>> Is it possible there are some hidden performance issues going on within
>> the JVM I need a special tool to see?
>>
>> Any help would be greatly appreciated.
>>
>> Thanks,
>> Scott
>>
>>
>>
>>