You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-dev@logging.apache.org by Guy Marom <ma...@gmail.com> on 2016/12/11 11:40:20 UTC
Counting logging events
Hello everyone,
I want to count all the logging events into our metrics system
(Prometheus). Our devs are using this to create alerts for Error/Fatal/Warn
events.
I gave up using filters because they catch all the events and not just the
ones that are actually written to the logger.
My current idea is to use a rewrite appender with my own rewrite policy
which will just increment the relevant counter and not modify the event.
The problem with this approach is again the filters, if an appender has a
filter defined then I will count events that end up discarded.
Is there some post-logging hook I can use? An event maybe? Any help will be
much appreciated...
Thanks,
Guy Marom
Re: Counting logging events
Posted by Guy Marom <ma...@gmail.com>.
Yes, that's my current plan.
Again, thanks for the help
On Dec 12, 2016 12:08 AM, "Matt Sicker" <bo...@gmail.com> wrote:
> Oh, I interpreted that backwards! Then the best layer to insert this logic
> at might be as an appender wrapper of some sort like he mentioned already.
>
> On 11 December 2016 at 16:05, Remko Popma <re...@gmail.com> wrote:
>
>> Matt,
>>
>> I think Guy wants to count at the Appender level after all filtering is
>> done.
>>
>> Sent from my iPhone
>>
>> On 12 Dec 2016, at 3:00, Matt Sicker <bo...@gmail.com> wrote:
>>
>> Do you mean something similar to this?
>> https://docs.oracle.com/javase/8/docs/api/java/util/logging/
>> LogRecord.html#getSequenceNumber--
>>
>> Although I think that has the same limitation as the number won't
>> increase if the log message is filtered before being appended (or in the
>> JUL vocabulary, the log record may be filtered before being handled).
>>
>> You could generate some Logger wrappers using the tool <
>> https://logging.apache.org/log4j/2.x/manual/customloglevels.html#CodeGen>
>> and add in an AtomicLong. Do note that if you're trying to count filtered
>> messages as well, you're going to lose the benefit of the no-ops when a
>> message is filtered. Also, this still wouldn't work if someone uses the
>> pattern:
>>
>> if (log.isDebugEnabled()) {
>> log.debug(...);
>> }
>>
>> On 11 December 2016 at 07:08, Guy Marom <ma...@gmail.com> wrote:
>>
>>> Hello again,
>>>
>>> Well, I think that both these options are not good for me.
>>> First I DO want to keep a separate track for each logger so the sequence
>>> number is no good.
>>>
>>> Second, using the a custom pattern or a lookup plugin both seem too
>>> intrusive to me. What if an appender already has a pattern or a lookup
>>> defined?.
>>>
>>> Unless someone comes up with a better I think I'll stick with my rewrite
>>> appender and just notify the user they cannot define a filter on the
>>> appender if they want the metrics to reflect reality.
>>>
>>> Thanks a lot for the help,
>>>
>>> Guy
>>>
>>> On Sun, Dec 11, 2016 at 2:11 PM, Guy Marom <ma...@gmail.com> wrote:
>>>
>>>> Thanks, I'll take a look at this suggestion!
>>>>
>>>> On Sun, Dec 11, 2016 at 1:56 PM, Remko Popma <re...@gmail.com>
>>>> wrote:
>>>>
>>>>> One option is the SequenceNumber
>>>>> <https://github.com/apache/logging-log4j2/blob/76d78fe9a4adecebd9805d051a1606b2ac70ccd0/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/SequenceNumberPatternConverter.java>
>>>>> pattern converter %sn in PatternLayout
>>>>> <https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout>
>>>>> .
>>>>> This uses a static counter, so if you need to keep separate track of
>>>>> multiple appenders this may not be useful.
>>>>> What you could do then is create a custom pattern converter plugin or
>>>>> lookup plugin.
>>>>>
>>>>> On Sun, Dec 11, 2016 at 8:40 PM, Guy Marom <ma...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello everyone,
>>>>>> I want to count all the logging events into our metrics system
>>>>>> (Prometheus). Our devs are using this to create alerts for Error/Fatal/Warn
>>>>>> events.
>>>>>>
>>>>>> I gave up using filters because they catch all the events and not
>>>>>> just the ones that are actually written to the logger.
>>>>>>
>>>>>> My current idea is to use a rewrite appender with my own rewrite
>>>>>> policy which will just increment the relevant counter and not modify the
>>>>>> event.
>>>>>> The problem with this approach is again the filters, if an appender
>>>>>> has a filter defined then I will count events that end up discarded.
>>>>>>
>>>>>> Is there some post-logging hook I can use? An event maybe? Any help
>>>>>> will be much appreciated...
>>>>>> Thanks,
>>>>>> Guy Marom
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Matt Sicker <bo...@gmail.com>
>>
>>
>
>
> --
> Matt Sicker <bo...@gmail.com>
>
Re: Counting logging events
Posted by Matt Sicker <bo...@gmail.com>.
Oh, I interpreted that backwards! Then the best layer to insert this logic
at might be as an appender wrapper of some sort like he mentioned already.
On 11 December 2016 at 16:05, Remko Popma <re...@gmail.com> wrote:
> Matt,
>
> I think Guy wants to count at the Appender level after all filtering is
> done.
>
> Sent from my iPhone
>
> On 12 Dec 2016, at 3:00, Matt Sicker <bo...@gmail.com> wrote:
>
> Do you mean something similar to this?
> https://docs.oracle.com/javase/8/docs/api/java/util/
> logging/LogRecord.html#getSequenceNumber--
>
> Although I think that has the same limitation as the number won't increase
> if the log message is filtered before being appended (or in the JUL
> vocabulary, the log record may be filtered before being handled).
>
> You could generate some Logger wrappers using the tool <
> https://logging.apache.org/log4j/2.x/manual/customloglevels.html#CodeGen>
> and add in an AtomicLong. Do note that if you're trying to count filtered
> messages as well, you're going to lose the benefit of the no-ops when a
> message is filtered. Also, this still wouldn't work if someone uses the
> pattern:
>
> if (log.isDebugEnabled()) {
> log.debug(...);
> }
>
> On 11 December 2016 at 07:08, Guy Marom <ma...@gmail.com> wrote:
>
>> Hello again,
>>
>> Well, I think that both these options are not good for me.
>> First I DO want to keep a separate track for each logger so the sequence
>> number is no good.
>>
>> Second, using the a custom pattern or a lookup plugin both seem too
>> intrusive to me. What if an appender already has a pattern or a lookup
>> defined?.
>>
>> Unless someone comes up with a better I think I'll stick with my rewrite
>> appender and just notify the user they cannot define a filter on the
>> appender if they want the metrics to reflect reality.
>>
>> Thanks a lot for the help,
>>
>> Guy
>>
>> On Sun, Dec 11, 2016 at 2:11 PM, Guy Marom <ma...@gmail.com> wrote:
>>
>>> Thanks, I'll take a look at this suggestion!
>>>
>>> On Sun, Dec 11, 2016 at 1:56 PM, Remko Popma <re...@gmail.com>
>>> wrote:
>>>
>>>> One option is the SequenceNumber
>>>> <https://github.com/apache/logging-log4j2/blob/76d78fe9a4adecebd9805d051a1606b2ac70ccd0/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/SequenceNumberPatternConverter.java>
>>>> pattern converter %sn in PatternLayout
>>>> <https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout>
>>>> .
>>>> This uses a static counter, so if you need to keep separate track of
>>>> multiple appenders this may not be useful.
>>>> What you could do then is create a custom pattern converter plugin or
>>>> lookup plugin.
>>>>
>>>> On Sun, Dec 11, 2016 at 8:40 PM, Guy Marom <ma...@gmail.com> wrote:
>>>>
>>>>> Hello everyone,
>>>>> I want to count all the logging events into our metrics system
>>>>> (Prometheus). Our devs are using this to create alerts for Error/Fatal/Warn
>>>>> events.
>>>>>
>>>>> I gave up using filters because they catch all the events and not just
>>>>> the ones that are actually written to the logger.
>>>>>
>>>>> My current idea is to use a rewrite appender with my own rewrite
>>>>> policy which will just increment the relevant counter and not modify the
>>>>> event.
>>>>> The problem with this approach is again the filters, if an appender
>>>>> has a filter defined then I will count events that end up discarded.
>>>>>
>>>>> Is there some post-logging hook I can use? An event maybe? Any help
>>>>> will be much appreciated...
>>>>> Thanks,
>>>>> Guy Marom
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> Matt Sicker <bo...@gmail.com>
>
>
--
Matt Sicker <bo...@gmail.com>
Re: Counting logging events
Posted by Remko Popma <re...@gmail.com>.
Matt,
I think Guy wants to count at the Appender level after all filtering is done.
Sent from my iPhone
> On 12 Dec 2016, at 3:00, Matt Sicker <bo...@gmail.com> wrote:
>
> Do you mean something similar to this?
> https://docs.oracle.com/javase/8/docs/api/java/util/logging/LogRecord.html#getSequenceNumber--
>
> Although I think that has the same limitation as the number won't increase if the log message is filtered before being appended (or in the JUL vocabulary, the log record may be filtered before being handled).
>
> You could generate some Logger wrappers using the tool <https://logging.apache.org/log4j/2.x/manual/customloglevels.html#CodeGen> and add in an AtomicLong. Do note that if you're trying to count filtered messages as well, you're going to lose the benefit of the no-ops when a message is filtered. Also, this still wouldn't work if someone uses the pattern:
>
> if (log.isDebugEnabled()) {
> log.debug(...);
> }
>
>> On 11 December 2016 at 07:08, Guy Marom <ma...@gmail.com> wrote:
>> Hello again,
>>
>> Well, I think that both these options are not good for me.
>> First I DO want to keep a separate track for each logger so the sequence number is no good.
>>
>> Second, using the a custom pattern or a lookup plugin both seem too intrusive to me. What if an appender already has a pattern or a lookup defined?.
>>
>> Unless someone comes up with a better I think I'll stick with my rewrite appender and just notify the user they cannot define a filter on the appender if they want the metrics to reflect reality.
>>
>> Thanks a lot for the help,
>>
>> Guy
>>
>>> On Sun, Dec 11, 2016 at 2:11 PM, Guy Marom <ma...@gmail.com> wrote:
>>> Thanks, I'll take a look at this suggestion!
>>>
>>>> On Sun, Dec 11, 2016 at 1:56 PM, Remko Popma <re...@gmail.com> wrote:
>>>> One option is the SequenceNumber pattern converter %sn in PatternLayout.
>>>> This uses a static counter, so if you need to keep separate track of multiple appenders this may not be useful.
>>>> What you could do then is create a custom pattern converter plugin or lookup plugin.
>>>>
>>>>> On Sun, Dec 11, 2016 at 8:40 PM, Guy Marom <ma...@gmail.com> wrote:
>>>>> Hello everyone,
>>>>> I want to count all the logging events into our metrics system (Prometheus). Our devs are using this to create alerts for Error/Fatal/Warn events.
>>>>>
>>>>> I gave up using filters because they catch all the events and not just the ones that are actually written to the logger.
>>>>>
>>>>> My current idea is to use a rewrite appender with my own rewrite policy which will just increment the relevant counter and not modify the event.
>>>>> The problem with this approach is again the filters, if an appender has a filter defined then I will count events that end up discarded.
>>>>>
>>>>> Is there some post-logging hook I can use? An event maybe? Any help will be much appreciated...
>>>>> Thanks,
>>>>> Guy Marom
>>>>
>>>
>>
>
>
>
> --
> Matt Sicker <bo...@gmail.com>
Re: Counting logging events
Posted by Matt Sicker <bo...@gmail.com>.
Do you mean something similar to this?
https://docs.oracle.com/javase/8/docs/api/java/util/logging/LogRecord.html#getSequenceNumber--
Although I think that has the same limitation as the number won't increase
if the log message is filtered before being appended (or in the JUL
vocabulary, the log record may be filtered before being handled).
You could generate some Logger wrappers using the tool <
https://logging.apache.org/log4j/2.x/manual/customloglevels.html#CodeGen>
and add in an AtomicLong. Do note that if you're trying to count filtered
messages as well, you're going to lose the benefit of the no-ops when a
message is filtered. Also, this still wouldn't work if someone uses the
pattern:
if (log.isDebugEnabled()) {
log.debug(...);
}
On 11 December 2016 at 07:08, Guy Marom <ma...@gmail.com> wrote:
> Hello again,
>
> Well, I think that both these options are not good for me.
> First I DO want to keep a separate track for each logger so the sequence
> number is no good.
>
> Second, using the a custom pattern or a lookup plugin both seem too
> intrusive to me. What if an appender already has a pattern or a lookup
> defined?.
>
> Unless someone comes up with a better I think I'll stick with my rewrite
> appender and just notify the user they cannot define a filter on the
> appender if they want the metrics to reflect reality.
>
> Thanks a lot for the help,
>
> Guy
>
> On Sun, Dec 11, 2016 at 2:11 PM, Guy Marom <ma...@gmail.com> wrote:
>
>> Thanks, I'll take a look at this suggestion!
>>
>> On Sun, Dec 11, 2016 at 1:56 PM, Remko Popma <re...@gmail.com>
>> wrote:
>>
>>> One option is the SequenceNumber
>>> <https://github.com/apache/logging-log4j2/blob/76d78fe9a4adecebd9805d051a1606b2ac70ccd0/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/SequenceNumberPatternConverter.java>
>>> pattern converter %sn in PatternLayout
>>> <https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout>
>>> .
>>> This uses a static counter, so if you need to keep separate track of
>>> multiple appenders this may not be useful.
>>> What you could do then is create a custom pattern converter plugin or
>>> lookup plugin.
>>>
>>> On Sun, Dec 11, 2016 at 8:40 PM, Guy Marom <ma...@gmail.com> wrote:
>>>
>>>> Hello everyone,
>>>> I want to count all the logging events into our metrics system
>>>> (Prometheus). Our devs are using this to create alerts for Error/Fatal/Warn
>>>> events.
>>>>
>>>> I gave up using filters because they catch all the events and not just
>>>> the ones that are actually written to the logger.
>>>>
>>>> My current idea is to use a rewrite appender with my own rewrite policy
>>>> which will just increment the relevant counter and not modify the event.
>>>> The problem with this approach is again the filters, if an appender has
>>>> a filter defined then I will count events that end up discarded.
>>>>
>>>> Is there some post-logging hook I can use? An event maybe? Any help
>>>> will be much appreciated...
>>>> Thanks,
>>>> Guy Marom
>>>>
>>>
>>>
>>
>
--
Matt Sicker <bo...@gmail.com>
Re: Counting logging events
Posted by Guy Marom <ma...@gmail.com>.
Hello again,
Well, I think that both these options are not good for me.
First I DO want to keep a separate track for each logger so the sequence
number is no good.
Second, using the a custom pattern or a lookup plugin both seem too
intrusive to me. What if an appender already has a pattern or a lookup
defined?.
Unless someone comes up with a better I think I'll stick with my rewrite
appender and just notify the user they cannot define a filter on the
appender if they want the metrics to reflect reality.
Thanks a lot for the help,
Guy
On Sun, Dec 11, 2016 at 2:11 PM, Guy Marom <ma...@gmail.com> wrote:
> Thanks, I'll take a look at this suggestion!
>
> On Sun, Dec 11, 2016 at 1:56 PM, Remko Popma <re...@gmail.com>
> wrote:
>
>> One option is the SequenceNumber
>> <https://github.com/apache/logging-log4j2/blob/76d78fe9a4adecebd9805d051a1606b2ac70ccd0/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/SequenceNumberPatternConverter.java>
>> pattern converter %sn in PatternLayout
>> <https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout>.
>> This uses a static counter, so if you need to keep separate track of
>> multiple appenders this may not be useful.
>> What you could do then is create a custom pattern converter plugin or
>> lookup plugin.
>>
>> On Sun, Dec 11, 2016 at 8:40 PM, Guy Marom <ma...@gmail.com> wrote:
>>
>>> Hello everyone,
>>> I want to count all the logging events into our metrics system
>>> (Prometheus). Our devs are using this to create alerts for Error/Fatal/Warn
>>> events.
>>>
>>> I gave up using filters because they catch all the events and not just
>>> the ones that are actually written to the logger.
>>>
>>> My current idea is to use a rewrite appender with my own rewrite policy
>>> which will just increment the relevant counter and not modify the event.
>>> The problem with this approach is again the filters, if an appender has
>>> a filter defined then I will count events that end up discarded.
>>>
>>> Is there some post-logging hook I can use? An event maybe? Any help will
>>> be much appreciated...
>>> Thanks,
>>> Guy Marom
>>>
>>
>>
>
Re: Counting logging events
Posted by Guy Marom <ma...@gmail.com>.
Thanks, I'll take a look at this suggestion!
On Sun, Dec 11, 2016 at 1:56 PM, Remko Popma <re...@gmail.com> wrote:
> One option is the SequenceNumber
> <https://github.com/apache/logging-log4j2/blob/76d78fe9a4adecebd9805d051a1606b2ac70ccd0/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/SequenceNumberPatternConverter.java>
> pattern converter %sn in PatternLayout
> <https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout>.
> This uses a static counter, so if you need to keep separate track of
> multiple appenders this may not be useful.
> What you could do then is create a custom pattern converter plugin or
> lookup plugin.
>
> On Sun, Dec 11, 2016 at 8:40 PM, Guy Marom <ma...@gmail.com> wrote:
>
>> Hello everyone,
>> I want to count all the logging events into our metrics system
>> (Prometheus). Our devs are using this to create alerts for Error/Fatal/Warn
>> events.
>>
>> I gave up using filters because they catch all the events and not just
>> the ones that are actually written to the logger.
>>
>> My current idea is to use a rewrite appender with my own rewrite policy
>> which will just increment the relevant counter and not modify the event.
>> The problem with this approach is again the filters, if an appender has a
>> filter defined then I will count events that end up discarded.
>>
>> Is there some post-logging hook I can use? An event maybe? Any help will
>> be much appreciated...
>> Thanks,
>> Guy Marom
>>
>
>
Re: Counting logging events
Posted by Remko Popma <re...@gmail.com>.
One option is the SequenceNumber
<https://github.com/apache/logging-log4j2/blob/76d78fe9a4adecebd9805d051a1606b2ac70ccd0/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/SequenceNumberPatternConverter.java>
pattern converter %sn in PatternLayout
<https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout>.
This uses a static counter, so if you need to keep separate track of
multiple appenders this may not be useful.
What you could do then is create a custom pattern converter plugin or
lookup plugin.
On Sun, Dec 11, 2016 at 8:40 PM, Guy Marom <ma...@gmail.com> wrote:
> Hello everyone,
> I want to count all the logging events into our metrics system
> (Prometheus). Our devs are using this to create alerts for Error/Fatal/Warn
> events.
>
> I gave up using filters because they catch all the events and not just the
> ones that are actually written to the logger.
>
> My current idea is to use a rewrite appender with my own rewrite policy
> which will just increment the relevant counter and not modify the event.
> The problem with this approach is again the filters, if an appender has a
> filter defined then I will count events that end up discarded.
>
> Is there some post-logging hook I can use? An event maybe? Any help will
> be much appreciated...
> Thanks,
> Guy Marom
>