You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Mike Thomsen <mi...@gmail.com> on 2023/07/06 14:28:06 UTC

NiFi not rolling logs

We are using the stock configuration, and have noticed that we have a lot
of nifi-app* logs that are well beyond the historic data cap of 30 days in
logback.xml; some of those logs go back to April. We also have a bunch of 0
byte nifi-user logs and some of the other logs are 0 bytes as well. It
looks like logback is rotating based on time, but isn't cleaning up. Is
this expected behavior or a problem with the configuration?

Thanks,

Mike

Re: NiFi not rolling logs

Posted by Lars Winderling <la...@posteo.de>.
David,
great recommendations, thanks!
Best, Lars

On 23-07-10 15:59, David Handermann wrote:
> It is worth noting that the totalSizeCap can be added at any time to 
> an existing NiFi deployment, it is not necessary to wait for an 
> updated release.
>
> Also related, the nifi-deprecation.log default setting includes a 100 
> MB totalSizeCap to avoid excessive repetition of deprecation messages, 
> and that has worked well for several releases.
>
> Regards,
> David Handermann
>
> On Sun, Jul 9, 2023 at 3:26 PM Mike Thomsen <mi...@gmail.com> 
> wrote:
>
>     The totalcapsize feature will help a lot from what I’ve seen.
>
>     Sent from my iPhone
>
>>     On Jul 8, 2023, at 9:26 AM, Lars Winderling
>>     <la...@posteo.de> wrote:
>>
>>     
>>     Hi Mike,
>>
>>     thanks for the advice. Our NiFi instances are running for week,
>>     if not months. Often times until the next update, so the startup
>>     option will bring much benefit, I fear, or am I mistaken. But
>>     looking forward to 1.23!
>>
>>
>>     On 8 July 2023 13:40:15 CEST, Mike Thomsen
>>     <mi...@gmail.com> wrote:
>>
>>         Lars,
>>
>>         You should also experiment with cleanHistoryOnStart. I did
>>         some experimentation this morning where I set the maxHistory
>>         to 1 (1 day vs the default of 30 which is 30 days), created a
>>         few fake log files from previous days and NiFi immediately
>>         cleared out those "old files" on startup. I have a Jira
>>         ticket up to fix this for 1.x and 2.x and will likely have it
>>         up today. Should definitely be ready for 1.23
>>
>>         On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling
>>         <la...@posteo.de> wrote:
>>
>>             Dear NiFiers, we have been bugged so much by overflowing
>>             logfiles, and nothing has ever helped. I thought it was
>>             just my lack of skills...especially when NiFi has some
>>             issues and keeps on spilling stacktraces with high
>>             frequency to disk, it eats up space quickly. I have
>>             created cronjobs that rotate logs every minute iff
>>             required, and when almost no space is left, it simply
>>             deletes old files. Will try totalCapSize etc. Thank you
>>             for the pointers! Best, Lars
>>
>>
>>             On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed"
>>             <jm...@gmail.com> wrote:
>>
>>                 Hi
>>
>>                 Please have a look at this old jira:
>>                 https://issues.apache.org/jira/browse/NIFI-2203
>>                 I have had issues where a processor create a log
>>                 message ever 10ms resulting in the disk is being
>>                 full. For me it seems like the maxHistory settings
>>                 only effect how many files defined by the rolling
>>                 patten to be kept. If you have defined it like this:
>>                 <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern>
>>                 MaxHistory only effect the days not the increments
>>                 file %i per day. So you can stille have thousands of
>>                 files in one day.
>>                 The totalSizeCap will delete the oldes files if the
>>                 total size hits the cap settings.
>>
>>                 The totalSizeCap have been added in the logback.xml
>>                 file for nifi-registry where it has been added inside
>>                 the rollingPolicy section. I cound not get it to work
>>                 inside the rollingPolicy section in nifi but just
>>                 added in appender section. See my comment in the
>>                 jira: https://issues.apache.org/jira/browse/NIFI-2203
>>
>>                 Kind regards
>>                 Jens M. Kofoed
>>
>>                 Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen
>>                 <mi...@gmail.com>:
>>
>>                     Yeah, I'm working through some of it where I have
>>                     time. I plan to have a Jira up this weekend. I'm
>>                     wondering, though, if we shouldn't consider a
>>                     spike for switching to log4j2 in 2.X because I
>>                     saw a lot of complaints about logback being
>>                     inconsistent in honoring its settings.
>>
>>                     On Fri, Jul 7, 2023 at 10:19 PM Joe Witt
>>                     <jo...@gmail.com> wrote:
>>
>>                         Hmmmm. Interesting.  Can you capture these
>>                         bits of fun in a jira?
>>
>>                         Thanks
>>
>>                         On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen
>>                         <mi...@gmail.com> wrote:
>>
>>                             After doing some research, it appears
>>                             that <maxFileSize/> is a wonky setting
>>                             WRT how well it's honored by logback. I
>>                             let a GenerateFlowFile > LogAttribute
>>                             flow run for a long time, and it just
>>                             kept filling up. When I added
>>                             <totalSizeCap/> that appeared to force
>>                             expected behavior on total log size. We
>>                             might want to add the following:
>>
>>                             <cleanHistoryOnStart>true</cleanHistoryOnStart>
>>                             <totalSizeCap>50GB</totalSizeCap>
>>
>>                             On Fri, Jul 7, 2023 at 11:33 AM Michael
>>                             Moser <mo...@gmail.com> wrote:
>>
>>                                 Hi Mike,
>>
>>                                 You aren't alone in experiencing
>>                                 this.  I think logback uses a pattern
>>                                 matcher on filename to discover files
>>                                 to delete.  If "something" happens
>>                                 which causes a gap in the date
>>                                 pattern, then the matcher will then
>>                                 fail to pick up and delete files on
>>                                 the other side of that gap.
>>
>>                                 Regards,
>>                                 -- Mike M
>>
>>
>>                                 On Thu, Jul 6, 2023 at 10:28 AM Mike
>>                                 Thomsen <mi...@gmail.com> wrote:
>>
>>                                     We are using the stock
>>                                     configuration, and have noticed
>>                                     that we have a lot of nifi-app*
>>                                     logs that are well beyond the
>>                                     historic data cap of 30 days in
>>                                     logback.xml; some of those logs
>>                                     go back to April. We also have a
>>                                     bunch of 0 byte nifi-user logs
>>                                     and some of the other logs are 0
>>                                     bytes as well. It looks like
>>                                     logback is rotating based on
>>                                     time, but isn't cleaning up. Is
>>                                     this expected behavior or a
>>                                     problem with the configuration?
>>
>>                                     Thanks,
>>
>>                                     Mike
>>


Re: NiFi not rolling logs

Posted by David Handermann <ex...@apache.org>.
It is worth noting that the totalSizeCap can be added at any time to an
existing NiFi deployment, it is not necessary to wait for an updated
release.

Also related, the nifi-deprecation.log default setting includes a 100 MB
totalSizeCap to avoid excessive repetition of deprecation messages, and
that has worked well for several releases.

Regards,
David Handermann

On Sun, Jul 9, 2023 at 3:26 PM Mike Thomsen <mi...@gmail.com> wrote:

> The totalcapsize feature will help a lot from what I’ve seen.
>
> Sent from my iPhone
>
> On Jul 8, 2023, at 9:26 AM, Lars Winderling <la...@posteo.de>
> wrote:
>
> 
> Hi Mike,
>
> thanks for the advice. Our NiFi instances are running for week, if not
> months. Often times until the next update, so the startup option will bring
> much benefit, I fear, or am I mistaken. But looking forward to 1.23!
>
>
> On 8 July 2023 13:40:15 CEST, Mike Thomsen <mi...@gmail.com> wrote:
>
>> Lars,
>>
>> You should also experiment with cleanHistoryOnStart. I did some
>> experimentation this morning where I set the maxHistory to 1 (1 day vs the
>> default of 30 which is 30 days), created a few fake log files from previous
>> days and NiFi immediately cleared out those "old files" on startup. I have
>> a Jira ticket up to fix this for 1.x and 2.x and will likely have it up
>> today. Should definitely be ready for 1.23
>>
>> On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling <la...@posteo.de>
>> wrote:
>>
>>> Dear NiFiers, we have been bugged so much by overflowing logfiles, and
>>> nothing has ever helped. I thought it was just my lack of
>>> skills...especially when NiFi has some issues and keeps on spilling
>>> stacktraces with high frequency to disk, it eats up space quickly. I have
>>> created cronjobs that rotate logs every minute iff required, and when
>>> almost no space is left, it simply deletes old files. Will try totalCapSize
>>> etc. Thank you for the pointers! Best, Lars
>>>
>>>
>>> On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" <jm...@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> Please have a look at this old jira:
>>>> https://issues.apache.org/jira/browse/NIFI-2203
>>>> I have had issues where a processor create a log message ever 10ms
>>>> resulting in the disk is being full. For me it seems like the maxHistory
>>>> settings only effect how many files defined by the rolling patten to be
>>>> kept. If you have defined it like this:
>>>>
>>>> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern>
>>>> MaxHistory only effect the days not the increments file %i per day. So
>>>> you can stille have thousands of files in one day.
>>>> The totalSizeCap will delete the oldes files if the total size hits
>>>> the cap settings.
>>>>
>>>> The totalSizeCap have been added in the logback.xml file for
>>>> nifi-registry where it has been added inside the rollingPolicy section. I
>>>> cound not get it to work inside the rollingPolicy section in nifi but just
>>>> added in appender section. See my comment in the jira:
>>>> https://issues.apache.org/jira/browse/NIFI-2203
>>>>
>>>> Kind regards
>>>> Jens M. Kofoed
>>>>
>>>> Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen <
>>>> mikerthomsen@gmail.com>:
>>>>
>>>>> Yeah, I'm working through some of it where I have time. I plan to have
>>>>> a Jira up this weekend. I'm wondering, though, if we shouldn't consider a
>>>>> spike for switching to log4j2 in 2.X because I saw a lot of complaints
>>>>> about logback being inconsistent in honoring its settings.
>>>>>
>>>>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <jo...@gmail.com> wrote:
>>>>>
>>>>>> Hmmmm.  Interesting.  Can you capture these bits of fun in a jira?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mi...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> After doing some research, it appears that <maxFileSize/> is a wonky
>>>>>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
>>>>>>> LogAttribute flow run for a long time, and it just kept filling up. When I
>>>>>>> added <totalSizeCap/> that appeared to force expected behavior on total log
>>>>>>> size. We might want to add the following:
>>>>>>>
>>>>>>> <cleanHistoryOnStart>true</cleanHistoryOnStart>
>>>>>>> <totalSizeCap>50GB</totalSizeCap>
>>>>>>>
>>>>>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <mo...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Mike,
>>>>>>>>
>>>>>>>> You aren't alone in experiencing this.  I think logback uses a
>>>>>>>> pattern matcher on filename to discover files to delete.  If "something"
>>>>>>>> happens which causes a gap in the date pattern, then the matcher will then
>>>>>>>> fail to pick up and delete files on the other side of that gap.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> -- Mike M
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <
>>>>>>>> mikerthomsen@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> We are using the stock configuration, and have noticed that we
>>>>>>>>> have a lot of nifi-app* logs that are well beyond the historic data cap of
>>>>>>>>> 30 days in logback.xml; some of those logs go back to April. We also have a
>>>>>>>>> bunch of 0 byte nifi-user logs and some of the other logs are 0 bytes as
>>>>>>>>> well. It looks like logback is rotating based on time, but isn't cleaning
>>>>>>>>> up. Is this expected behavior or a problem with the configuration?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>

Re: NiFi not rolling logs

Posted by Mike Thomsen <mi...@gmail.com>.
The totalcapsize feature will help a lot from what I’ve seen.  
  

Sent from my iPhone

  

> On Jul 8, 2023, at 9:26 AM, Lars Winderling <la...@posteo.de>
> wrote:  
>  
>

> 
>
> Hi Mike,  
>  
> thanks for the advice. Our NiFi instances are running for week, if not
> months. Often times until the next update, so the startup option will bring
> much benefit, I fear, or am I mistaken. But looking forward to 1.23!
>
>  
>  
>
>
> On 8 July 2023 13:40:15 CEST, Mike Thomsen <mi...@gmail.com> wrote:
>

>> Lars,

>>

>>  
>
>>

>> You should also experiment with cleanHistoryOnStart. I did some
experimentation this morning where I set the maxHistory to 1 (1 day vs the
default of 30 which is 30 days), created a few fake log files from previous
days and NiFi immediately cleared out those "old files" on startup. I have a
Jira ticket up to fix this for 1.x and 2.x and will likely have it up today.
Should definitely be ready for 1.23

>>

>>  
>
>>

>> On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling
<[lars.winderling@posteo.de](mailto:lars.winderling@posteo.de)> wrote:  
>
>>

>>>  __

>>>

>>> Dear NiFiers, we have been bugged so much by overflowing logfiles, and
nothing has ever helped. I thought it was just my lack of skills...especially
when NiFi has some issues and keeps on spilling stacktraces with high
frequency to disk, it eats up space quickly. I have created cronjobs that
rotate logs every minute iff required, and when almost no space is left, it
simply deletes old files. Will try totalCapSize etc. Thank you for the
pointers! Best, Lars

>>>

>>>  
>  
>
>>>

>>> On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed"
<[jmkofoed.ube@gmail.com](mailto:jmkofoed.ube@gmail.com)> wrote:

>>>

>>>> Hi

>>>>

>>>>  
>
>>>>

>>>> Please have a look at this old jira:
<https://issues.apache.org/jira/browse/NIFI-2203>

>>>>

>>>> I have had issues where a processor create a log message ever 10ms
resulting in the disk is being full. For me it seems like the maxHistory
settings only effect how many files defined by the rolling patten to be kept.
If you have defined it like this:  
> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-
> app%d{yyyy-MM-dd}.%i.log</fileNamePattern>  
> MaxHistory only effect the days not the increments file %i per day. So you
> can stille have thousands of files in one day.
>>>>

>>>> The totalSizeCap will delete the oldes files if the total size hits the
cap settings.  
>
>>>>

>>>>  
>
>>>>

>>>> The totalSizeCap have been added in the logback.xml file for nifi-
registry where it has been added inside the rollingPolicy section. I cound not
get it to work inside the rollingPolicy section in nifi but just added in
appender section. See my comment in the jira:
<https://issues.apache.org/jira/browse/NIFI-2203>

>>>>

>>>>  
>
>>>>

>>>> Kind regards

>>>>

>>>> Jens M. Kofoed

>>>>

>>>>  
>
>>>>

>>>> Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen
<[mikerthomsen@gmail.com](mailto:mikerthomsen@gmail.com)>:  
>
>>>>

>>>>> Yeah, I'm working through some of it where I have time. I plan to have a
Jira up this weekend. I'm wondering, though, if we shouldn't consider a spike
for switching to log4j2 in 2.X because I saw a lot of complaints about logback
being inconsistent in honoring its settings.

>>>>>

>>>>>  
>
>>>>>

>>>>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt
<[joe.witt@gmail.com](mailto:joe.witt@gmail.com)> wrote:  
>
>>>>>

>>>>>> Hmmmm.  Interesting.  Can you capture these bits of fun in a jira?

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> Thanks

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen
<[mikerthomsen@gmail.com](mailto:mikerthomsen@gmail.com)> wrote:  
>
>>>>>>

>>>>>>> After doing some research, it appears that <maxFileSize/> is a wonky
setting WRT how well it's honored by logback. I let a GenerateFlowFile >
LogAttribute flow run for a long time, and it just kept filling up. When I
added <totalSizeCap/> that appeared to force expected behavior on total log
size. We might want to add the following:

>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> <cleanHistoryOnStart>true</cleanHistoryOnStart>

>>>>>>>

>>>>>>> <totalSizeCap>50GB</totalSizeCap>

>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser
<[moser.mw@gmail.com](mailto:moser.mw@gmail.com)> wrote:  
>
>>>>>>>

>>>>>>>> Hi Mike,

>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> You aren't alone in experiencing this.  I think logback uses a
pattern matcher on filename to discover files to delete.  If "something"
happens which causes a gap in the date pattern, then the matcher will then
fail to pick up and delete files on the other side of that gap.

>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> Regards,

>>>>>>>>

>>>>>>>> \-- Mike M

>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen
<[mikerthomsen@gmail.com](mailto:mikerthomsen@gmail.com)> wrote:  
>
>>>>>>>>

>>>>>>>>> We are using the stock configuration, and have noticed that we have
a lot of nifi-app* logs that are well beyond the historic data cap of 30 days
in logback.xml; some of those logs go back to April. We also have a bunch of 0
byte nifi-user logs and some of the other logs are 0 bytes as well. It looks
like logback is rotating based on time, but isn't cleaning up. Is this
expected behavior or a problem with the configuration?

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Thanks,

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Mike


Re: NiFi not rolling logs

Posted by Lars Winderling <la...@posteo.de>.
Hi Mike,

thanks for the advice. Our NiFi instances are running for week, if not months. Often times until the next update, so the startup option will bring much benefit, I fear, or am I mistaken. But looking forward to 1.23!

On 8 July 2023 13:40:15 CEST, Mike Thomsen <mi...@gmail.com> wrote:
>Lars,
>
>You should also experiment with cleanHistoryOnStart. I did some
>experimentation this morning where I set the maxHistory to 1 (1 day vs the
>default of 30 which is 30 days), created a few fake log files from previous
>days and NiFi immediately cleared out those "old files" on startup. I have
>a Jira ticket up to fix this for 1.x and 2.x and will likely have it up
>today. Should definitely be ready for 1.23
>
>On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling <la...@posteo.de>
>wrote:
>
>> Dear NiFiers, we have been bugged so much by overflowing logfiles, and
>> nothing has ever helped. I thought it was just my lack of
>> skills...especially when NiFi has some issues and keeps on spilling
>> stacktraces with high frequency to disk, it eats up space quickly. I have
>> created cronjobs that rotate logs every minute iff required, and when
>> almost no space is left, it simply deletes old files. Will try totalCapSize
>> etc. Thank you for the pointers! Best, Lars
>>
>>
>> On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" <jm...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> Please have a look at this old jira:
>>> https://issues.apache.org/jira/browse/NIFI-2203
>>> I have had issues where a processor create a log message ever 10ms
>>> resulting in the disk is being full. For me it seems like the maxHistory
>>> settings only effect how many files defined by the rolling patten to be
>>> kept. If you have defined it like this:
>>>
>>> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern>
>>> MaxHistory only effect the days not the increments file %i per day. So
>>> you can stille have thousands of files in one day.
>>> The totalSizeCap will delete the oldes files if the total size hits the
>>> cap settings.
>>>
>>> The totalSizeCap have been added in the logback.xml file for
>>> nifi-registry where it has been added inside the rollingPolicy section. I
>>> cound not get it to work inside the rollingPolicy section in nifi but just
>>> added in appender section. See my comment in the jira:
>>> https://issues.apache.org/jira/browse/NIFI-2203
>>>
>>> Kind regards
>>> Jens M. Kofoed
>>>
>>> Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen <
>>> mikerthomsen@gmail.com>:
>>>
>>>> Yeah, I'm working through some of it where I have time. I plan to have a
>>>> Jira up this weekend. I'm wondering, though, if we shouldn't consider a
>>>> spike for switching to log4j2 in 2.X because I saw a lot of complaints
>>>> about logback being inconsistent in honoring its settings.
>>>>
>>>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <jo...@gmail.com> wrote:
>>>>
>>>>> Hmmmm.  Interesting.  Can you capture these bits of fun in a jira?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> After doing some research, it appears that <maxFileSize/> is a wonky
>>>>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
>>>>>> LogAttribute flow run for a long time, and it just kept filling up. When I
>>>>>> added <totalSizeCap/> that appeared to force expected behavior on total log
>>>>>> size. We might want to add the following:
>>>>>>
>>>>>> <cleanHistoryOnStart>true</cleanHistoryOnStart>
>>>>>> <totalSizeCap>50GB</totalSizeCap>
>>>>>>
>>>>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <mo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Mike,
>>>>>>>
>>>>>>> You aren't alone in experiencing this.  I think logback uses a
>>>>>>> pattern matcher on filename to discover files to delete.  If "something"
>>>>>>> happens which causes a gap in the date pattern, then the matcher will then
>>>>>>> fail to pick up and delete files on the other side of that gap.
>>>>>>>
>>>>>>> Regards,
>>>>>>> -- Mike M
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mi...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> We are using the stock configuration, and have noticed that we have
>>>>>>>> a lot of nifi-app* logs that are well beyond the historic data cap of 30
>>>>>>>> days in logback.xml; some of those logs go back to April. We also have a
>>>>>>>> bunch of 0 byte nifi-user logs and some of the other logs are 0 bytes as
>>>>>>>> well. It looks like logback is rotating based on time, but isn't cleaning
>>>>>>>> up. Is this expected behavior or a problem with the configuration?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>

Re: NiFi not rolling logs

Posted by Mike Thomsen <mi...@gmail.com>.
Lars,

You should also experiment with cleanHistoryOnStart. I did some
experimentation this morning where I set the maxHistory to 1 (1 day vs the
default of 30 which is 30 days), created a few fake log files from previous
days and NiFi immediately cleared out those "old files" on startup. I have
a Jira ticket up to fix this for 1.x and 2.x and will likely have it up
today. Should definitely be ready for 1.23

On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling <la...@posteo.de>
wrote:

> Dear NiFiers, we have been bugged so much by overflowing logfiles, and
> nothing has ever helped. I thought it was just my lack of
> skills...especially when NiFi has some issues and keeps on spilling
> stacktraces with high frequency to disk, it eats up space quickly. I have
> created cronjobs that rotate logs every minute iff required, and when
> almost no space is left, it simply deletes old files. Will try totalCapSize
> etc. Thank you for the pointers! Best, Lars
>
>
> On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" <jm...@gmail.com>
> wrote:
>
>> Hi
>>
>> Please have a look at this old jira:
>> https://issues.apache.org/jira/browse/NIFI-2203
>> I have had issues where a processor create a log message ever 10ms
>> resulting in the disk is being full. For me it seems like the maxHistory
>> settings only effect how many files defined by the rolling patten to be
>> kept. If you have defined it like this:
>>
>> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern>
>> MaxHistory only effect the days not the increments file %i per day. So
>> you can stille have thousands of files in one day.
>> The totalSizeCap will delete the oldes files if the total size hits the
>> cap settings.
>>
>> The totalSizeCap have been added in the logback.xml file for
>> nifi-registry where it has been added inside the rollingPolicy section. I
>> cound not get it to work inside the rollingPolicy section in nifi but just
>> added in appender section. See my comment in the jira:
>> https://issues.apache.org/jira/browse/NIFI-2203
>>
>> Kind regards
>> Jens M. Kofoed
>>
>> Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen <
>> mikerthomsen@gmail.com>:
>>
>>> Yeah, I'm working through some of it where I have time. I plan to have a
>>> Jira up this weekend. I'm wondering, though, if we shouldn't consider a
>>> spike for switching to log4j2 in 2.X because I saw a lot of complaints
>>> about logback being inconsistent in honoring its settings.
>>>
>>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <jo...@gmail.com> wrote:
>>>
>>>> Hmmmm.  Interesting.  Can you capture these bits of fun in a jira?
>>>>
>>>> Thanks
>>>>
>>>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mi...@gmail.com>
>>>> wrote:
>>>>
>>>>> After doing some research, it appears that <maxFileSize/> is a wonky
>>>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
>>>>> LogAttribute flow run for a long time, and it just kept filling up. When I
>>>>> added <totalSizeCap/> that appeared to force expected behavior on total log
>>>>> size. We might want to add the following:
>>>>>
>>>>> <cleanHistoryOnStart>true</cleanHistoryOnStart>
>>>>> <totalSizeCap>50GB</totalSizeCap>
>>>>>
>>>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Mike,
>>>>>>
>>>>>> You aren't alone in experiencing this.  I think logback uses a
>>>>>> pattern matcher on filename to discover files to delete.  If "something"
>>>>>> happens which causes a gap in the date pattern, then the matcher will then
>>>>>> fail to pick up and delete files on the other side of that gap.
>>>>>>
>>>>>> Regards,
>>>>>> -- Mike M
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mi...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> We are using the stock configuration, and have noticed that we have
>>>>>>> a lot of nifi-app* logs that are well beyond the historic data cap of 30
>>>>>>> days in logback.xml; some of those logs go back to April. We also have a
>>>>>>> bunch of 0 byte nifi-user logs and some of the other logs are 0 bytes as
>>>>>>> well. It looks like logback is rotating based on time, but isn't cleaning
>>>>>>> up. Is this expected behavior or a problem with the configuration?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>

Re: NiFi not rolling logs

Posted by Lars Winderling <la...@posteo.de>.
Dear NiFiers, we have been bugged so much by overflowing logfiles, and nothing has ever helped. I thought it was just my lack of skills...especially when NiFi has some issues and keeps on spilling stacktraces with high frequency to disk, it eats up space quickly. I have created cronjobs that rotate logs every minute iff required, and when almost no space is left, it simply deletes old files. Will try totalCapSize etc. Thank you for the pointers! Best, Lars

On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" <jm...@gmail.com> wrote:
>Hi
>
>Please have a look at this old jira:
>https://issues.apache.org/jira/browse/NIFI-2203
>I have had issues where a processor create a log message ever 10ms
>resulting in the disk is being full. For me it seems like the maxHistory
>settings only effect how many files defined by the rolling patten to be
>kept. If you have defined it like this:
><fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern>
>MaxHistory only effect the days not the increments file %i per day. So you
>can stille have thousands of files in one day.
>The totalSizeCap will delete the oldes files if the total size hits the cap
>settings.
>
>The totalSizeCap have been added in the logback.xml file for nifi-registry
>where it has been added inside the rollingPolicy section. I cound not get
>it to work inside the rollingPolicy section in nifi but just added
>in appender section. See my comment in the jira:
>https://issues.apache.org/jira/browse/NIFI-2203
>
>Kind regards
>Jens M. Kofoed
>
>Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen <mi...@gmail.com>:
>
>> Yeah, I'm working through some of it where I have time. I plan to have a
>> Jira up this weekend. I'm wondering, though, if we shouldn't consider a
>> spike for switching to log4j2 in 2.X because I saw a lot of complaints
>> about logback being inconsistent in honoring its settings.
>>
>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <jo...@gmail.com> wrote:
>>
>>> Hmmmm.  Interesting.  Can you capture these bits of fun in a jira?
>>>
>>> Thanks
>>>
>>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mi...@gmail.com>
>>> wrote:
>>>
>>>> After doing some research, it appears that <maxFileSize/> is a wonky
>>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
>>>> LogAttribute flow run for a long time, and it just kept filling up. When I
>>>> added <totalSizeCap/> that appeared to force expected behavior on total log
>>>> size. We might want to add the following:
>>>>
>>>> <cleanHistoryOnStart>true</cleanHistoryOnStart>
>>>> <totalSizeCap>50GB</totalSizeCap>
>>>>
>>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <mo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Mike,
>>>>>
>>>>> You aren't alone in experiencing this.  I think logback uses a pattern
>>>>> matcher on filename to discover files to delete.  If "something" happens
>>>>> which causes a gap in the date pattern, then the matcher will then fail to
>>>>> pick up and delete files on the other side of that gap.
>>>>>
>>>>> Regards,
>>>>> -- Mike M
>>>>>
>>>>>
>>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> We are using the stock configuration, and have noticed that we have a
>>>>>> lot of nifi-app* logs that are well beyond the historic data cap of 30 days
>>>>>> in logback.xml; some of those logs go back to April. We also have a bunch
>>>>>> of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. It
>>>>>> looks like logback is rotating based on time, but isn't cleaning up. Is
>>>>>> this expected behavior or a problem with the configuration?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>

Re: NiFi not rolling logs

Posted by "Jens M. Kofoed" <jm...@gmail.com>.
Hi

Please have a look at this old jira:
https://issues.apache.org/jira/browse/NIFI-2203
I have had issues where a processor create a log message ever 10ms
resulting in the disk is being full. For me it seems like the maxHistory
settings only effect how many files defined by the rolling patten to be
kept. If you have defined it like this:
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern>
MaxHistory only effect the days not the increments file %i per day. So you
can stille have thousands of files in one day.
The totalSizeCap will delete the oldes files if the total size hits the cap
settings.

The totalSizeCap have been added in the logback.xml file for nifi-registry
where it has been added inside the rollingPolicy section. I cound not get
it to work inside the rollingPolicy section in nifi but just added
in appender section. See my comment in the jira:
https://issues.apache.org/jira/browse/NIFI-2203

Kind regards
Jens M. Kofoed

Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen <mi...@gmail.com>:

> Yeah, I'm working through some of it where I have time. I plan to have a
> Jira up this weekend. I'm wondering, though, if we shouldn't consider a
> spike for switching to log4j2 in 2.X because I saw a lot of complaints
> about logback being inconsistent in honoring its settings.
>
> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <jo...@gmail.com> wrote:
>
>> Hmmmm.  Interesting.  Can you capture these bits of fun in a jira?
>>
>> Thanks
>>
>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> After doing some research, it appears that <maxFileSize/> is a wonky
>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
>>> LogAttribute flow run for a long time, and it just kept filling up. When I
>>> added <totalSizeCap/> that appeared to force expected behavior on total log
>>> size. We might want to add the following:
>>>
>>> <cleanHistoryOnStart>true</cleanHistoryOnStart>
>>> <totalSizeCap>50GB</totalSizeCap>
>>>
>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <mo...@gmail.com>
>>> wrote:
>>>
>>>> Hi Mike,
>>>>
>>>> You aren't alone in experiencing this.  I think logback uses a pattern
>>>> matcher on filename to discover files to delete.  If "something" happens
>>>> which causes a gap in the date pattern, then the matcher will then fail to
>>>> pick up and delete files on the other side of that gap.
>>>>
>>>> Regards,
>>>> -- Mike M
>>>>
>>>>
>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mi...@gmail.com>
>>>> wrote:
>>>>
>>>>> We are using the stock configuration, and have noticed that we have a
>>>>> lot of nifi-app* logs that are well beyond the historic data cap of 30 days
>>>>> in logback.xml; some of those logs go back to April. We also have a bunch
>>>>> of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. It
>>>>> looks like logback is rotating based on time, but isn't cleaning up. Is
>>>>> this expected behavior or a problem with the configuration?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike
>>>>>
>>>>

Re: NiFi not rolling logs

Posted by Mike Thomsen <mi...@gmail.com>.
Yeah, I'm working through some of it where I have time. I plan to have a
Jira up this weekend. I'm wondering, though, if we shouldn't consider a
spike for switching to log4j2 in 2.X because I saw a lot of complaints
about logback being inconsistent in honoring its settings.

On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <jo...@gmail.com> wrote:

> Hmmmm.  Interesting.  Can you capture these bits of fun in a jira?
>
> Thanks
>
> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> After doing some research, it appears that <maxFileSize/> is a wonky
>> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
>> LogAttribute flow run for a long time, and it just kept filling up. When I
>> added <totalSizeCap/> that appeared to force expected behavior on total log
>> size. We might want to add the following:
>>
>> <cleanHistoryOnStart>true</cleanHistoryOnStart>
>> <totalSizeCap>50GB</totalSizeCap>
>>
>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <mo...@gmail.com> wrote:
>>
>>> Hi Mike,
>>>
>>> You aren't alone in experiencing this.  I think logback uses a pattern
>>> matcher on filename to discover files to delete.  If "something" happens
>>> which causes a gap in the date pattern, then the matcher will then fail to
>>> pick up and delete files on the other side of that gap.
>>>
>>> Regards,
>>> -- Mike M
>>>
>>>
>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mi...@gmail.com>
>>> wrote:
>>>
>>>> We are using the stock configuration, and have noticed that we have a
>>>> lot of nifi-app* logs that are well beyond the historic data cap of 30 days
>>>> in logback.xml; some of those logs go back to April. We also have a bunch
>>>> of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. It
>>>> looks like logback is rotating based on time, but isn't cleaning up. Is
>>>> this expected behavior or a problem with the configuration?
>>>>
>>>> Thanks,
>>>>
>>>> Mike
>>>>
>>>

Re: NiFi not rolling logs

Posted by Joe Witt <jo...@gmail.com>.
Hmmmm.  Interesting.  Can you capture these bits of fun in a jira?

Thanks

On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mi...@gmail.com> wrote:

> After doing some research, it appears that <maxFileSize/> is a wonky
> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
> LogAttribute flow run for a long time, and it just kept filling up. When I
> added <totalSizeCap/> that appeared to force expected behavior on total log
> size. We might want to add the following:
>
> <cleanHistoryOnStart>true</cleanHistoryOnStart>
> <totalSizeCap>50GB</totalSizeCap>
>
> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <mo...@gmail.com> wrote:
>
>> Hi Mike,
>>
>> You aren't alone in experiencing this.  I think logback uses a pattern
>> matcher on filename to discover files to delete.  If "something" happens
>> which causes a gap in the date pattern, then the matcher will then fail to
>> pick up and delete files on the other side of that gap.
>>
>> Regards,
>> -- Mike M
>>
>>
>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> We are using the stock configuration, and have noticed that we have a
>>> lot of nifi-app* logs that are well beyond the historic data cap of 30 days
>>> in logback.xml; some of those logs go back to April. We also have a bunch
>>> of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. It
>>> looks like logback is rotating based on time, but isn't cleaning up. Is
>>> this expected behavior or a problem with the configuration?
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>

Re: NiFi not rolling logs

Posted by Mike Thomsen <mi...@gmail.com>.
After doing some research, it appears that <maxFileSize/> is a wonky
setting WRT how well it's honored by logback. I let a GenerateFlowFile >
LogAttribute flow run for a long time, and it just kept filling up. When I
added <totalSizeCap/> that appeared to force expected behavior on total log
size. We might want to add the following:

<cleanHistoryOnStart>true</cleanHistoryOnStart>
<totalSizeCap>50GB</totalSizeCap>

On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <mo...@gmail.com> wrote:

> Hi Mike,
>
> You aren't alone in experiencing this.  I think logback uses a pattern
> matcher on filename to discover files to delete.  If "something" happens
> which causes a gap in the date pattern, then the matcher will then fail to
> pick up and delete files on the other side of that gap.
>
> Regards,
> -- Mike M
>
>
> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> We are using the stock configuration, and have noticed that we have a lot
>> of nifi-app* logs that are well beyond the historic data cap of 30 days in
>> logback.xml; some of those logs go back to April. We also have a bunch of 0
>> byte nifi-user logs and some of the other logs are 0 bytes as well. It
>> looks like logback is rotating based on time, but isn't cleaning up. Is
>> this expected behavior or a problem with the configuration?
>>
>> Thanks,
>>
>> Mike
>>
>

Re: NiFi not rolling logs

Posted by Michael Moser <mo...@gmail.com>.
Hi Mike,

You aren't alone in experiencing this.  I think logback uses a pattern
matcher on filename to discover files to delete.  If "something" happens
which causes a gap in the date pattern, then the matcher will then fail to
pick up and delete files on the other side of that gap.

Regards,
-- Mike M


On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mi...@gmail.com> wrote:

> We are using the stock configuration, and have noticed that we have a lot
> of nifi-app* logs that are well beyond the historic data cap of 30 days in
> logback.xml; some of those logs go back to April. We also have a bunch of 0
> byte nifi-user logs and some of the other logs are 0 bytes as well. It
> looks like logback is rotating based on time, but isn't cleaning up. Is
> this expected behavior or a problem with the configuration?
>
> Thanks,
>
> Mike
>