Logging SPI?

[Niclas Hedhman <ni...@hedhman.org>]

Everyone,

We have relatively little logging going on in the Core Runtime and also in
most libraries/extensions. Part of the reason is that there is not enough
consensus on what the best logging approach is, and the most used ones are
all horrible.

I suggest that we introduce a Logging Extension SPI, and in good Polygene
tradition define our direct needs from top/down, and then allow for
pluggable extensions to deal with this as they see fit.

I think there is no need for a "LoggerFactory" (I could be wrong), but that
we simply introduce a few types to be injectable in @Structure scope, each
dedicated for the purpose, instead of the generic "log" that is prevalent
in logging systems deriving from Log4j. There are also control needs, as I
prefer to not have external configuration as the primary controller for
what is used or not, since that will effectively become implementation
specific. Things like log namespace and enable/disable must exist in the
SPI, possibly other things. And I think the extension of controls should
also be a prescribed mechanism, maybe have additional common configuration
options in the SPI, even if they don't apply to all implementations, e.g.
Formatting which is very common, but not for a log that writes entities.

I see 4 distinct subsystems within this SPI;

    Trace -  dealing with tracing of methods. Since we own the method
invocations, this will be relatively easy to do throughout a Polygene
application, and this injectable type is about controlling what is traced,
where reporting goes and what type of timing should be applied. Here is the
greatest level of integration into the Core Runtime.
In the library-logging, this is enabled in bootstrap, but I have since
changed my mind and think we need it to be controllable in runtime.

    Debug - We all need debug information, and be able to let the code tell
us what is going on. The output is dedicated for developers, and is
disabled in Application.production mode. This is one of the most common
use-cases, especially for us developers. *@Structure Debug debug;* and
reasonable methods on that type is all we need. We should also try to use
lambdas here, to get around the *if( debug.isEnabled() ){}*, which is ugly
and *debug.debug( () -> "here!" + param );* would look better and don't
incur the cost of message construction if not enabled.

    LogBook - Better name might needed. The "Captain's Log" is all about
the information that production operations personnel is interested in.
Important state changes, events and other relevant information is to be
communicated here.

    Audit  -  Many business applications have Audit requirements. I think
we should have dedicated support for this, rather than expecting people to
model the whole infrastructure from scratch.


As usual, I think Polygene should provide all these things in much more
type-safe fashion and customizable types, than what we have in "traditional
logging" systems.

By making this an SPI, and by having Noop implementation(s) as default, it
is much easier for people to default to using this, rather than SLF4J,
which frankly is horrible (albeit prolific in the enterprise).


In the implementation side, there will be room for both traditional
solutions as well as innovation. Oh, yeah... We should try to figure out if
we can redirect the existing slf4j, log4j, jdk, commons-logging and what
not APIs to go through our SPI, to provide a unified channel for Polygene
applications.


WDYAT?


Cheers
-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: Logging SPI?

[Niclas Hedhman <ni...@hedhman.org>]

As usual, good feedback...

On Thu, Nov 30, 2017 at 5:09 AM, Kent Sølvsten <ke...@gmail.com>
wrote:

> On Sat, Nov 25, 2017 at 4:41 AM, Niclas Hedhman <ni...@hedhman.org>
> wrote:
>
> > I think there is no need for a "LoggerFactory" (I could be wrong), but
> that
> > we simply introduce a few types to be injectable in @Structure scope,
> each
> > dedicated for the purpose, instead of the generic "log" that is prevalent
> > in logging systems deriving from Log4j.
>
>
> Very much agree that we should avoid another Factory. But I actually would
> prefer the types to @Service instead of @Structure - more like eg.
> EntityStores.
>

The reason I wanted @Structure is to capture the caller's module, rather
than the service's. And that the implementation ends up being a @Service,
just like UnitOfWork ends up delegating to services.

But, when thinking about it, UnitOfWorkFactory is now optionally a Service.
So, I should check how the @Structure semantics are handle or if there is a
design flaw, and if it is done correctly, perhaps use the same mechanism.



> There are also control needs, as I
> > prefer to not have external configuration as the primary controller for
> > what is used or not, since that will effectively become implementation
> > specific. Things like log namespace and enable/disable must exist in the
> > SPI, possibly other things.
>
>
> Some sort of Service activation/passivation?
>

Well, one use-case I have in mind is that code should be able to enable
TRACE for one or more methods programmatically, to do measurements in
production selectively. Maybe a Concern does a TRACE every 1000 requests or
so and capture that data in a timeseries.

Once there is an interface for that, then other use-cases are likely to
surface soon enough, maybe time sensitive debug capture.



> > And I think the extension of controls should
> > also be a prescribed mechanism, maybe have additional common
> configuration
> > options in the SPI, even if they don't apply to all implementations, e.g.
> > Formatting which is very common, but not for a log that writes entities.
> >
> > I see 4 distinct subsystems within this SPI;
> >
> >     Trace -  dealing with tracing of methods. Since we own the method
> > invocations, this will be relatively easy to do throughout a Polygene
> > application, and this injectable type is about controlling what is
> traced,
> > where reporting goes and what type of timing should be applied. Here is
> the
> > greatest level of integration into the Core Runtime.
> > In the library-logging, this is enabled in bootstrap, but I have since
> > changed my mind and think we need it to be controllable in runtime.
> >
>
> An extra concern added automatically to everything?
>

Maybe (Side-Effect, though). Let's see if usecases shows up that can't
easily be done with fragments.


> >     Debug - We all need debug information, and be able to let the code
> tell
> > us what is going on. The output is dedicated for developers, and is
> > disabled in Application.production mode. This is one of the most common
> > use-cases, especially for us developers. *@Structure Debug debug;* and
> > reasonable methods on that type is all we need. We should also try to use
> > lambdas here, to get around the *if( debug.isEnabled() ){}*, which is
> ugly
> > and *debug.debug( () -> "here!" + param );* would look better and don't
> > incur the cost of message construction if not enabled.
> >
>
> Note that logger#isDebugEnabled may not be necessary at all, since eg.
> SLF4J has support for something like
>
> Object entry = new SomeObject();
> logger.debug("The entry is {}.", entry);
>

Yeah, unless the entry is expensive to create instead...



> >     LogBook - Better name might needed. The "Captain's Log" is all about
> > the information that production operations personnel is interested in.
> > Important state changes, events and other relevant information is to be
> > communicated here.
> >
>
> A special case of a more generic ApplicationEvent concept?
>

Yeah, you might be on to something here. Is it time to do ApplicationEvent
design first? 4 visible event buses, one in the module, one in the layer
and one for each above/below layer? That sounds quite innovative, and only
possible with our structure enforcement. Possibly introduce custom-made
"global" eventbuses as well. Each of these ties into a implementations
similar to EntityStores. Should kick up a new thread on this...



> >     Audit  -  Many business applications have Audit requirements. I think
> > we should have dedicated support for this, rather than expecting people
> to
> > model the whole infrastructure from scratch.
> >
>
> This is quite a hot topic in the EU area a.t.m. (GDPR). A good solution
> here would really be awesome!
>

Well, I would guess that a political committee would not come up with
something awesome, but instead enforce something bad and utterly
complicated onto some poor industries. But hey, I could be proven wrong.

I think for us, this is a matter of an @Audit annotation with a bunch of
fields, such as "name", "reference", "description" and the whole
invocation, with method and arguments (possibly filtered out with another
annotation), to be passed to a service, with implementations taking care of
store and forwards.



> > In the implementation side, there will be room for both traditional
> > solutions as well as innovation. Oh, yeah... We should try to figure out
> if
> > we can redirect the existing slf4j, log4j, jdk, commons-logging and what
> > not APIs to go through our SPI, to provide a unified channel for Polygene
> > applications.
> >
>
> Not sure we should do this channelling except for the debug logging
> - I tend to think it is much better to let users migrate their code to use
> a more explicit API
> - and thus expose the @Service / @Structure to application code.
>

Well, preferably I want to disable the use of SLF4J and similar APIs in
Polygene application code. Not sure that we can/should manage that. The
usecase I had in mind was that external libraries are not going to change
to Polygene, and it is those I want to capture. Some better libraries, such
as Restlet, has a pluggable implementation, but most are hardcoded to one
or the other.


Cheers
-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: Logging SPI?

[Kent Sølvsten <ke...@gmail.com>]

Hi Niclas.

As always a lot of interesting thoughts. A few thoughts/comments below.


On Sat, Nov 25, 2017 at 4:41 AM, Niclas Hedhman <ni...@hedhman.org> wrote:

> Everyone,
>
> We have relatively little logging going on in the Core Runtime and also in
> most libraries/extensions. Part of the reason is that there is not enough
> consensus on what the best logging approach is, and the most used ones are
> all horrible.
>
> I suggest that we introduce a Logging Extension SPI, and in good Polygene
> tradition define our direct needs from top/down, and then allow for
> pluggable extensions to deal with this as they see fit.
>
> I think there is no need for a "LoggerFactory" (I could be wrong), but that
> we simply introduce a few types to be injectable in @Structure scope, each
> dedicated for the purpose, instead of the generic "log" that is prevalent
> in logging systems deriving from Log4j.


Very much agree that we should avoid another Factory. But I actually would
prefer the types to @Service instead of @Structure - more like eg.
EntityStores.
I find it more flexible to just use a ServiceFinder internally for finding
the implementation. We could consider using a ServiceReference internally
to allow for them being optional.

There are also control needs, as I
> prefer to not have external configuration as the primary controller for
> what is used or not, since that will effectively become implementation
> specific. Things like log namespace and enable/disable must exist in the
> SPI, possibly other things.


Some sort of Service activation/passivation?


> And I think the extension of controls should
> also be a prescribed mechanism, maybe have additional common configuration
> options in the SPI, even if they don't apply to all implementations, e.g.
> Formatting which is very common, but not for a log that writes entities.
>
> I see 4 distinct subsystems within this SPI;
>
>     Trace -  dealing with tracing of methods. Since we own the method
> invocations, this will be relatively easy to do throughout a Polygene
> application, and this injectable type is about controlling what is traced,
> where reporting goes and what type of timing should be applied. Here is the
> greatest level of integration into the Core Runtime.
> In the library-logging, this is enabled in bootstrap, but I have since
> changed my mind and think we need it to be controllable in runtime.
>

An extra concern added automatically to everything?

>
>     Debug - We all need debug information, and be able to let the code tell
> us what is going on. The output is dedicated for developers, and is
> disabled in Application.production mode. This is one of the most common
> use-cases, especially for us developers. *@Structure Debug debug;* and
> reasonable methods on that type is all we need. We should also try to use
> lambdas here, to get around the *if( debug.isEnabled() ){}*, which is ugly
> and *debug.debug( () -> "here!" + param );* would look better and don't
> incur the cost of message construction if not enabled.
>

Note that logger#isDebugEnabled may not be necessary at all, since eg.
SLF4J has support for something like

Object entry = new SomeObject();
logger.debug("The entry is {}.", entry);



>     LogBook - Better name might needed. The "Captain's Log" is all about
> the information that production operations personnel is interested in.
> Important state changes, events and other relevant information is to be
> communicated here.
>

A special case of a more generic ApplicationEvent concept?

>
>     Audit  -  Many business applications have Audit requirements. I think
> we should have dedicated support for this, rather than expecting people to
> model the whole infrastructure from scratch.
>

This is quite a hot topic in the EU area a.t.m. (GDPR). A good solution
here would really be awesome!

>
>
> As usual, I think Polygene should provide all these things in much more
> type-safe fashion and customizable types, than what we have in "traditional
> logging" systems.
>
> By making this an SPI, and by having Noop implementation(s) as default, it
> is much easier for people to default to using this, rather than SLF4J,
> which frankly is horrible (albeit prolific in the enterprise).
>
>
> In the implementation side, there will be room for both traditional
> solutions as well as innovation. Oh, yeah... We should try to figure out if
> we can redirect the existing slf4j, log4j, jdk, commons-logging and what
> not APIs to go through our SPI, to provide a unified channel for Polygene
> applications.
>

Not sure we should do this channelling except for the debug logging
- I tend to think it is much better to let users migrate their code to use
a more explicit API
- and thus expose the @Service / @Structure to application code.

>
>
> WDYAT?
>
>
> Cheers
> --
> Niclas Hedhman, Software Developer
> http://polygene.apache.org - New Energy for Java
>

/Kent