[DISCUSS] stop logging errors when named pipe is full

[Emanuele Rocca <em...@wikimedia.org>]

Hi,

on busy servers, using named pipes as a log destination is a useful way
to avoid wasting disk space and bandwidth [1]. However, Traffic Server
currently logs an error message when the buffer fills up because no
process is reading from the pipe.

On a test system, we've seen lots of "Failed to write log" entries in
such a scenario, with systemd-journal's CPU usage quickly going up and
disk space and bandwidth being wasted.

As not having any clients consuming log entries from the named pipe
unless needed for debugging purposes seems like a plausible use case, I
was wondering whether Traffic Server should stop logging an error when
writes fail with EPIPE (something along the lines of [2])?

[1] https://docs.trafficserver.apache.org/en/latest/admin-guide/logging/destinations.en.html#named-pipes
[2] https://github.com/ema/trafficserver/commit/1831100badaa6b9c72a80cde5a4c2966c94be634

Re: [DISCUSS] stop logging errors when named pipe is full

[Leif Hedstrom <zw...@apache.org>]

> On Sep 5, 2018, at 9:36 AM, Bryan Call <bc...@apache.org> wrote:
> 
> How about warning periodically or having a configuration option to disable the error message?


I think rate limiting warnings is a good idea. +1. Heck, I might even suggest that Warning() ought to have that built in :-).

I looked at the patch in #2 above, and it seems that could lose useful information? Presumably you might want to know that something has gone wrong (the missing reader) ? It’s not a totally unreadable patch, I’m just concerned that you are missing out on information that is useful.

Cheers,

— Leif

Re: [DISCUSS] stop logging errors when named pipe is full

[Bryan Call <bc...@apache.org>]

How about warning periodically or having a configuration option to disable the error message?

-Bryan



> On Sep 5, 2018, at 7:41 AM, Emanuele Rocca <em...@wikimedia.org> wrote:
> 
> Hi,
> 
> on busy servers, using named pipes as a log destination is a useful way
> to avoid wasting disk space and bandwidth [1]. However, Traffic Server
> currently logs an error message when the buffer fills up because no
> process is reading from the pipe.
> 
> On a test system, we've seen lots of "Failed to write log" entries in
> such a scenario, with systemd-journal's CPU usage quickly going up and
> disk space and bandwidth being wasted.
> 
> As not having any clients consuming log entries from the named pipe
> unless needed for debugging purposes seems like a plausible use case, I
> was wondering whether Traffic Server should stop logging an error when
> writes fail with EPIPE (something along the lines of [2])?
> 
> [1] https://docs.trafficserver.apache.org/en/latest/admin-guide/logging/destinations.en.html#named-pipes
> [2] https://github.com/ema/trafficserver/commit/1831100badaa6b9c72a80cde5a4c2966c94be634