You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-user@logging.apache.org by hm...@gmail.com on 2019/11/25 03:52:54 UTC
lookahead regex filter?
Hi all,
I am looking for a way to do a lookahead regex filter: ignore log lines, unless there is a match with a given pattern, in which case the N previous lines should be stored.
The RegexFilter filter doesn't seem to support this, or at least I haven't managed to make it work. (I would have been surprised to find it worked anyway)
Any tip on how to do this kind of thing?
Best,
--- Horacio
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: lookahead regex filter?
Posted by hm...@gmail.com.
Well, the requirements *that I imagined / gathered from a quick browse of the JIRA discussion* are
1. covering the case of having one full ring when an indefinitely long burst of triggers comes
2. producing a log with a monotonically-increasing timestamp
If we *assume* that draining one entry from the full ring takes as long as [swapping in another ring + storing into that ring], then you need as many ring entries as swappable rings. But if (for example) draining one entry took 10 times as long as the swap+store, then you would need 10N rings of N entries each.
This would work if you drain-until-trigger. One could also imagine keeping storing into the triggered ring while draining it to completion, but sounds like extra complexity that doesn't really give you anything new over the previous scenario.
My experience comes from embedded development on microcontrollers, in which circular buffers are the bread and butter and things tended to go (in my case at least) rather in lockstep between producers and consumers - so the N rings of N entries was kinda natural. Of course I have no idea how much this applies to Log4j, sorry for jumping in if not.
By the way, given that there were mentions of the Disruptor in JIRA: I wrote about the Disruptor vs "plain ring buffers" some time ago: https://hmijailblog.blogspot.com/2018/08/you-could-have-invented-lmax-disruptor.html <https://hmijailblog.blogspot.com/2018/08/you-could-have-invented-lmax-disruptor.html>
Best,
--- Horacio
> On 26 Nov 2019, at 1:52 am, Gary Gregory <ga...@gmail.com> wrote:
>
> Is there any reason why the number of ring buffers and the number of their
> entries should match?
>
> Gary
>
> On Mon, Nov 25, 2019 at 12:13 AM <hm...@gmail.com> wrote:
>
>> I'm sympathetic to the anxiety for getting everything designed right, but
>> at some point it's better to stop pondering possibilities and just get
>> something working ;).
>>
>> I have some experience in this kind of design space but 0 knowledge about
>> log4j internals, and my totally unrequested suggestion is that, since
>> probably simplicity / speed / garbage-free operation are valued, I'd go for
>> N ring buffers of N entries each, with N defined and everything allocated
>> at start-up time. Every new trigger swaps in a new ring buffer to be used
>> while the previous ring/s get drained. For the extreme case of multiple
>> triggers coming in a burst, you'd be swapping rings at every entry. Rings
>> are emptied until the triggering entry, it's the simplest (or only?) way to
>> have a well-defined result.
>>
>> For my immediate needs I guess I'll have to make do with some external
>> post-processing...
>> Thank you!
>>
>> Best,
>> --- Horacio
>>
>>> On 25 Nov 2019, at 2:47 pm, Ralph Goers <ra...@dslextreme.com>
>> wrote:
>>>
>>> See https://issues.apache.org/jira/browse/LOG4J2-1137 <
>> https://issues.apache.org/jira/browse/LOG4J2-1137>. It has been on my
>> todo list for quite a while. What I plan on implementing is a
>> “TriggeringAppender” where you can attach a custom trigger or triggers to
>> cause the event to fire. The problem I haven’t figure out is in capturing
>> and emitting the events. It should use a ring buffer but when the event
>> fires you don’t want to cause a pause while all the events are logged so
>> you would probably need more than one RingBuffer. And what if the trigger
>> fires again before the first trigger has logged all the events? I want to
>> figure out this behavior before I start writing it and I just haven’t had
>> the time to design it.
>>>
>>> Ralph
>>>
>>>> On Nov 24, 2019, at 8:52 PM, hmijail+log4j@gmail.com wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I am looking for a way to do a lookahead regex filter: ignore log
>> lines, unless there is a match with a given pattern, in which case the N
>> previous lines should be stored.
>>>>
>>>> The RegexFilter filter doesn't seem to support this, or at least I
>> haven't managed to make it work. (I would have been surprised to find it
>> worked anyway)
>>>>
>>>> Any tip on how to do this kind of thing?
>>>>
>>>> Best,
>>>> --- Horacio
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>
>>>>
>>>
>>
>>
Re: lookahead regex filter?
Posted by Gary Gregory <ga...@gmail.com>.
Is there any reason why the number of ring buffers and the number of their
entries should match?
Gary
On Mon, Nov 25, 2019 at 12:13 AM <hm...@gmail.com> wrote:
> I'm sympathetic to the anxiety for getting everything designed right, but
> at some point it's better to stop pondering possibilities and just get
> something working ;).
>
> I have some experience in this kind of design space but 0 knowledge about
> log4j internals, and my totally unrequested suggestion is that, since
> probably simplicity / speed / garbage-free operation are valued, I'd go for
> N ring buffers of N entries each, with N defined and everything allocated
> at start-up time. Every new trigger swaps in a new ring buffer to be used
> while the previous ring/s get drained. For the extreme case of multiple
> triggers coming in a burst, you'd be swapping rings at every entry. Rings
> are emptied until the triggering entry, it's the simplest (or only?) way to
> have a well-defined result.
>
> For my immediate needs I guess I'll have to make do with some external
> post-processing...
> Thank you!
>
> Best,
> --- Horacio
>
> > On 25 Nov 2019, at 2:47 pm, Ralph Goers <ra...@dslextreme.com>
> wrote:
> >
> > See https://issues.apache.org/jira/browse/LOG4J2-1137 <
> https://issues.apache.org/jira/browse/LOG4J2-1137>. It has been on my
> todo list for quite a while. What I plan on implementing is a
> “TriggeringAppender” where you can attach a custom trigger or triggers to
> cause the event to fire. The problem I haven’t figure out is in capturing
> and emitting the events. It should use a ring buffer but when the event
> fires you don’t want to cause a pause while all the events are logged so
> you would probably need more than one RingBuffer. And what if the trigger
> fires again before the first trigger has logged all the events? I want to
> figure out this behavior before I start writing it and I just haven’t had
> the time to design it.
> >
> > Ralph
> >
> >> On Nov 24, 2019, at 8:52 PM, hmijail+log4j@gmail.com wrote:
> >>
> >> Hi all,
> >>
> >> I am looking for a way to do a lookahead regex filter: ignore log
> lines, unless there is a match with a given pattern, in which case the N
> previous lines should be stored.
> >>
> >> The RegexFilter filter doesn't seem to support this, or at least I
> haven't managed to make it work. (I would have been surprised to find it
> worked anyway)
> >>
> >> Any tip on how to do this kind of thing?
> >>
> >> Best,
> >> --- Horacio
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> >> For additional commands, e-mail: log4j-user-help@logging.apache.org
> >>
> >>
> >
>
>
Re: lookahead regex filter?
Posted by hm...@gmail.com.
I'm sympathetic to the anxiety for getting everything designed right, but at some point it's better to stop pondering possibilities and just get something working ;).
I have some experience in this kind of design space but 0 knowledge about log4j internals, and my totally unrequested suggestion is that, since probably simplicity / speed / garbage-free operation are valued, I'd go for N ring buffers of N entries each, with N defined and everything allocated at start-up time. Every new trigger swaps in a new ring buffer to be used while the previous ring/s get drained. For the extreme case of multiple triggers coming in a burst, you'd be swapping rings at every entry. Rings are emptied until the triggering entry, it's the simplest (or only?) way to have a well-defined result.
For my immediate needs I guess I'll have to make do with some external post-processing...
Thank you!
Best,
--- Horacio
> On 25 Nov 2019, at 2:47 pm, Ralph Goers <ra...@dslextreme.com> wrote:
>
> See https://issues.apache.org/jira/browse/LOG4J2-1137 <https://issues.apache.org/jira/browse/LOG4J2-1137>. It has been on my todo list for quite a while. What I plan on implementing is a “TriggeringAppender” where you can attach a custom trigger or triggers to cause the event to fire. The problem I haven’t figure out is in capturing and emitting the events. It should use a ring buffer but when the event fires you don’t want to cause a pause while all the events are logged so you would probably need more than one RingBuffer. And what if the trigger fires again before the first trigger has logged all the events? I want to figure out this behavior before I start writing it and I just haven’t had the time to design it.
>
> Ralph
>
>> On Nov 24, 2019, at 8:52 PM, hmijail+log4j@gmail.com wrote:
>>
>> Hi all,
>>
>> I am looking for a way to do a lookahead regex filter: ignore log lines, unless there is a match with a given pattern, in which case the N previous lines should be stored.
>>
>> The RegexFilter filter doesn't seem to support this, or at least I haven't managed to make it work. (I would have been surprised to find it worked anyway)
>>
>> Any tip on how to do this kind of thing?
>>
>> Best,
>> --- Horacio
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>
>>
>
Re: lookahead regex filter?
Posted by Ralph Goers <ra...@dslextreme.com>.
See https://issues.apache.org/jira/browse/LOG4J2-1137 <https://issues.apache.org/jira/browse/LOG4J2-1137>. It has been on my todo list for quite a while. What I plan on implementing is a “TriggeringAppender” where you can attach a custom trigger or triggers to cause the event to fire. The problem I haven’t figure out is in capturing and emitting the events. It should use a ring buffer but when the event fires you don’t want to cause a pause while all the events are logged so you would probably need more than one RingBuffer. And what if the trigger fires again before the first trigger has logged all the events? I want to figure out this behavior before I start writing it and I just haven’t had the time to design it.
Ralph
> On Nov 24, 2019, at 8:52 PM, hmijail+log4j@gmail.com wrote:
>
> Hi all,
>
> I am looking for a way to do a lookahead regex filter: ignore log lines, unless there is a match with a given pattern, in which case the N previous lines should be stored.
>
> The RegexFilter filter doesn't seem to support this, or at least I haven't managed to make it work. (I would have been surprised to find it worked anyway)
>
> Any tip on how to do this kind of thing?
>
> Best,
> --- Horacio
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>