Logs and Disk space usage

[Devin Fisher <de...@perfectsearchcorp.com>]

I'm still early in the usage of Nifi at my company (no production usage
yet). But in developing some custom processors and other components I've
run both my dev machine and virtual machines out of disk space from verbose
logs. In all of these cases they have been my fault (turn debug level on
and left it running over the weekend and processor with a bug). But this
has left wondering what are people's experiences with the logging and what
are the best practice.

>From what I can tell Nifi uses logback (not much in the documentation about
it). And the default configuration for the app log is
a TimeBasedRollingPolicy that rolls the log every hour (or more often if
needed) and keeps logs for the last 30 hours.  Is this what most people are
using in production? Any issues?

I'm considering changing the configuration to a FixedWindowRollingPolicy.
With reasonable large files (100 MB) and a reasonable number of them (15).
I'm considering this because if there is something misbehaving I don't have
to worry about it filling my disk space. Thoughts about this approach?

Anyway, I'd love to hear what people are doing in production before I go
that way.

Devin

Re: Logs and Disk space usage

[Andre <an...@fucs.org>]

Devin,

I never had issues with Logs (they will not take too much space under
normal conditions) but I would suggest particular attention to:

1. Misbehaving processors may cause a flood error stacks that will quickly
fill logs (I only seen this happening with processors developed in-house or
outside the main codebase)

2. Content archives... Since version 0.3 (or 0.2) NiFi comes configured to
archive the content of flows for X hours (12 by default). I recently
noticed that one of my test boxes running a late snapshot of 0.4.1 was not
deleting the archives, causing the content_repository to balloon. Be
mindful to check if your archives are being deleted (as they should on
later versions).

Cheers

On Sat, Mar 26, 2016 at 9:52 AM, Devin Fisher <
devin.fisher@perfectsearchcorp.com> wrote:

> I'm still early in the usage of Nifi at my company (no production usage
> yet). But in developing some custom processors and other components I've
> run both my dev machine and virtual machines out of disk space from verbose
> logs. In all of these cases they have been my fault (turn debug level on
> and left it running over the weekend and processor with a bug). But this
> has left wondering what are people's experiences with the logging and what
> are the best practice.
>
> From what I can tell Nifi uses logback (not much in the documentation
> about it). And the default configuration for the app log is
> a TimeBasedRollingPolicy that rolls the log every hour (or more often if
> needed) and keeps logs for the last 30 hours.  Is this what most people are
> using in production? Any issues?
>
> I'm considering changing the configuration to a FixedWindowRollingPolicy.
> With reasonable large files (100 MB) and a reasonable number of them (15).
> I'm considering this because if there is something misbehaving I don't have
> to worry about it filling my disk space. Thoughts about this approach?
>
> Anyway, I'd love to hear what people are doing in production before I go
> that way.
>
> Devin
>

Re: Logs and Disk space usage

[Matthew Clarke <ma...@gmail.com>]

Devin,
      Make sure you are compressing your logs when they roll by adding a
.gz.  The logs will compress quite nicely and safe you considerable disk
space. As far as production goes, you want to maintaining enough log
history so that problem can be inventoried and resolved. In many cases the
default logger for NiFi is changed from INFO to WARN. This will also reduce
the amount of logging. The other thing you should be doing is winding logs
someplace other then where NiFi lives. If it should still fill that disk it
will have less of an impact on your NiFi application.

Matt
On Mar 25, 2016 6:52 PM, "Devin Fisher" <de...@perfectsearchcorp.com>
wrote:

> I'm still early in the usage of Nifi at my company (no production usage
> yet). But in developing some custom processors and other components I've
> run both my dev machine and virtual machines out of disk space from verbose
> logs. In all of these cases they have been my fault (turn debug level on
> and left it running over the weekend and processor with a bug). But this
> has left wondering what are people's experiences with the logging and what
> are the best practice.
>
> From what I can tell Nifi uses logback (not much in the documentation
> about it). And the default configuration for the app log is
> a TimeBasedRollingPolicy that rolls the log every hour (or more often if
> needed) and keeps logs for the last 30 hours.  Is this what most people are
> using in production? Any issues?
>
> I'm considering changing the configuration to a FixedWindowRollingPolicy.
> With reasonable large files (100 MB) and a reasonable number of them (15).
> I'm considering this because if there is something misbehaving I don't have
> to worry about it filling my disk space. Thoughts about this approach?
>
> Anyway, I'd love to hear what people are doing in production before I go
> that way.
>
> Devin
>