[LOG4J2-2803] Getting back to the plugin system rewrite

[Matt Sicker <bo...@gmail.com>]

Hey all, that itch is back to figure out how to complete my previous
attempt at the plugin system updates (started a little over a year ago
and put on pause at some point). I'm currently working on getting that
branch back up to working again with master (mostly needing to update
module-info files), and I'm hoping to map out and figure out the path
forward on completing it.

With that being said, one major aspect of this project will be
explaining how the 2.x plugin system works, how it relates to the 3.x
system, and the gist of how I'd like to restructure things. Does
anyone here have any preferences on how I share some docs about this?
I could record some videos that go over code and such, or I could
write things up blog style. Note that these docs would be focused at
fellow log4j developers and not necessarily the user manual. I still
plan to have proper API docs and manual pages for the system, but
those would be targeted for users writing their own extensions mainly.

Link for background info:
https://issues.apache.org/jira/browse/LOG4J2-2803

Re: [LOG4J2-2803] Getting back to the plugin system rewrite

[Ralph Goers <ra...@dslextreme.com>]

A couple of points:
1. I believe JsonTemplateLayout was recently modified to also use the plugin system.
2. Now that api, plugin and core are JPMS modules I’d like to modify all the others that
 could reasonably be made modules, make another pass at core in eliminating 
dependencies and hiding internal stuff and then push a release. I’m hoping we can do 
that by the end of the year. Hopefully your timelines will align with that.

Also, we have to take a look at where we have broken compatibility with 2.x and either 
add back stuff that was removed or document the breakage. Which we choose should 
be based on the pain it will cause users. I am sure we will have quite a few that would 
like to migrate to 3.0 but will have dependencies that are still relying on 2.x.

Ralph

> On Jun 5, 2021, at 1:34 PM, Matt Sicker <bo...@gmail.com> wrote:
> 
> I've updated https://issues.apache.org/jira/browse/LOG4J2-2803 with a
> better overview of the existing systems, the idea behind the new
> system, and some other details. I have some written notes about the
> state of the mean-bean-machine branch relating to what's left to
> integrate which is essentially the following:
> 
> * Format for bean metadata
> * Strategy for loading bean metadata
> * Annotation processor for generating bean metadata
> * Implementing a configuration scope
> * Transforming plugin attribute names into generated qualifier names
> based on configuration node location
> * Lazy initialization of required beans
> * Determine if and how bean managers should communicate (i.e., one
> bean manager per LoggerContext, or one per LoggerContextFactory?)
> * Refactor plugins to use new API
> 
> On Sat, 29 May 2021 at 20:58, Ralph Goers <ra...@dslextreme.com> wrote:
>> 
>> While you could write the ServiceLoader class by hand, I wouldn’t want anyone to.
>> That would be error prone.
>> 
>> I have though about creating a Maven plugin to generate the class, but all that
>> would do is replace the proc=only compile step, so there isn’t much point.
>> 
>> Ralph
>> 
>>> On May 29, 2021, at 5:40 PM, Ralph Goers <ra...@dslextreme.com> wrote:
>>> 
>>> We do need better docs on this. 3.0 is going to be a mixed bag. It will be easier
>>> for “normal” applications because plugins now use ServiceLoader instead of
>>> the Log4jPlugins.dat file. However, it is going to be more challenging for those
>>> developing plugins if they use the Java Platform Module System.
>>> 
>>> As one might imagine, the compiler now has a modulePath in addition to a
>>> classPath. When no module-info.java is present Maven will set things up to
>>> only use the classPath, which will work as normal. However, when a module-info.java
>>> file is present than Maven will configure the project and its tests to use the
>>> modulePath. By itself this isn’t a problem. However, the Java compiler ONLY
>>> looks for Annotation processors automatically on the classPath. This means
>>> anyone using JPMS while compiling plugins will have to explicitly configure
>>> the annotation processor.
>>> 
>>> There is a second issue when using JPMS. While our annotation processor
>>> generates the service class and service definition file, JPMS requires that the
>>> Log4jPlugins service class must be declared in module-info.java. This is
>>> somewhat of a problem because the compiler will fail if it finds the
>>> module-info.java references the Log4jPlugins service class because the
>>> annotation processor hasn’t generated it yet.
>>> 
>>> The only recourse to this is to do a proc=only compile that does not include
>>> the module-info.java class followed by a full compile with proc=none. Personally,
>>> I think this is a but in javac but I suspect it would be a major change to fix it.
>>> 
>>> I have meant to write a blog post on all of this basically to ask the question,
>>> “Is JPMS really worth it”?
>>> 
>>> Ralph
>>> 
>>>> On May 29, 2021, at 11:06 AM, Matt Sicker <bo...@gmail.com> wrote:
>>>> 
>>>> Hey all, that itch is back to figure out how to complete my previous
>>>> attempt at the plugin system updates (started a little over a year ago
>>>> and put on pause at some point). I'm currently working on getting that
>>>> branch back up to working again with master (mostly needing to update
>>>> module-info files), and I'm hoping to map out and figure out the path
>>>> forward on completing it.
>>>> 
>>>> With that being said, one major aspect of this project will be
>>>> explaining how the 2.x plugin system works, how it relates to the 3.x
>>>> system, and the gist of how I'd like to restructure things. Does
>>>> anyone here have any preferences on how I share some docs about this?
>>>> I could record some videos that go over code and such, or I could
>>>> write things up blog style. Note that these docs would be focused at
>>>> fellow log4j developers and not necessarily the user manual. I still
>>>> plan to have proper API docs and manual pages for the system, but
>>>> those would be targeted for users writing their own extensions mainly.
>>>> 
>>>> Link for background info:
>>>> https://issues.apache.org/jira/browse/LOG4J2-2803
>>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 

Re: [LOG4J2-2803] Getting back to the plugin system rewrite

[Matt Sicker <bo...@gmail.com>]

I've updated https://issues.apache.org/jira/browse/LOG4J2-2803 with a
better overview of the existing systems, the idea behind the new
system, and some other details. I have some written notes about the
state of the mean-bean-machine branch relating to what's left to
integrate which is essentially the following:

* Format for bean metadata
* Strategy for loading bean metadata
* Annotation processor for generating bean metadata
* Implementing a configuration scope
* Transforming plugin attribute names into generated qualifier names
based on configuration node location
* Lazy initialization of required beans
* Determine if and how bean managers should communicate (i.e., one
bean manager per LoggerContext, or one per LoggerContextFactory?)
* Refactor plugins to use new API

On Sat, 29 May 2021 at 20:58, Ralph Goers <ra...@dslextreme.com> wrote:
>
> While you could write the ServiceLoader class by hand, I wouldn’t want anyone to.
> That would be error prone.
>
> I have though about creating a Maven plugin to generate the class, but all that
> would do is replace the proc=only compile step, so there isn’t much point.
>
> Ralph
>
> > On May 29, 2021, at 5:40 PM, Ralph Goers <ra...@dslextreme.com> wrote:
> >
> > We do need better docs on this. 3.0 is going to be a mixed bag. It will be easier
> > for “normal” applications because plugins now use ServiceLoader instead of
> > the Log4jPlugins.dat file. However, it is going to be more challenging for those
> > developing plugins if they use the Java Platform Module System.
> >
> > As one might imagine, the compiler now has a modulePath in addition to a
> > classPath. When no module-info.java is present Maven will set things up to
> > only use the classPath, which will work as normal. However, when a module-info.java
> > file is present than Maven will configure the project and its tests to use the
> > modulePath. By itself this isn’t a problem. However, the Java compiler ONLY
> > looks for Annotation processors automatically on the classPath. This means
> > anyone using JPMS while compiling plugins will have to explicitly configure
> > the annotation processor.
> >
> > There is a second issue when using JPMS. While our annotation processor
> > generates the service class and service definition file, JPMS requires that the
> > Log4jPlugins service class must be declared in module-info.java. This is
> > somewhat of a problem because the compiler will fail if it finds the
> > module-info.java references the Log4jPlugins service class because the
> > annotation processor hasn’t generated it yet.
> >
> > The only recourse to this is to do a proc=only compile that does not include
> > the module-info.java class followed by a full compile with proc=none. Personally,
> > I think this is a but in javac but I suspect it would be a major change to fix it.
> >
> > I have meant to write a blog post on all of this basically to ask the question,
> > “Is JPMS really worth it”?
> >
> > Ralph
> >
> >> On May 29, 2021, at 11:06 AM, Matt Sicker <bo...@gmail.com> wrote:
> >>
> >> Hey all, that itch is back to figure out how to complete my previous
> >> attempt at the plugin system updates (started a little over a year ago
> >> and put on pause at some point). I'm currently working on getting that
> >> branch back up to working again with master (mostly needing to update
> >> module-info files), and I'm hoping to map out and figure out the path
> >> forward on completing it.
> >>
> >> With that being said, one major aspect of this project will be
> >> explaining how the 2.x plugin system works, how it relates to the 3.x
> >> system, and the gist of how I'd like to restructure things. Does
> >> anyone here have any preferences on how I share some docs about this?
> >> I could record some videos that go over code and such, or I could
> >> write things up blog style. Note that these docs would be focused at
> >> fellow log4j developers and not necessarily the user manual. I still
> >> plan to have proper API docs and manual pages for the system, but
> >> those would be targeted for users writing their own extensions mainly.
> >>
> >> Link for background info:
> >> https://issues.apache.org/jira/browse/LOG4J2-2803
> >>
> >
> >
> >
>
>

Re: [LOG4J2-2803] Getting back to the plugin system rewrite

[Ralph Goers <ra...@dslextreme.com>]

While you could write the ServiceLoader class by hand, I wouldn’t want anyone to. 
That would be error prone.

I have though about creating a Maven plugin to generate the class, but all that 
would do is replace the proc=only compile step, so there isn’t much point.

Ralph

> On May 29, 2021, at 5:40 PM, Ralph Goers <ra...@dslextreme.com> wrote:
> 
> We do need better docs on this. 3.0 is going to be a mixed bag. It will be easier 
> for “normal” applications because plugins now use ServiceLoader instead of 
> the Log4jPlugins.dat file. However, it is going to be more challenging for those 
> developing plugins if they use the Java Platform Module System.
> 
> As one might imagine, the compiler now has a modulePath in addition to a 
> classPath. When no module-info.java is present Maven will set things up to 
> only use the classPath, which will work as normal. However, when a module-info.java 
> file is present than Maven will configure the project and its tests to use the 
> modulePath. By itself this isn’t a problem. However, the Java compiler ONLY 
> looks for Annotation processors automatically on the classPath. This means 
> anyone using JPMS while compiling plugins will have to explicitly configure 
> the annotation processor. 
> 
> There is a second issue when using JPMS. While our annotation processor 
> generates the service class and service definition file, JPMS requires that the 
> Log4jPlugins service class must be declared in module-info.java. This is 
> somewhat of a problem because the compiler will fail if it finds the 
> module-info.java references the Log4jPlugins service class because the 
> annotation processor hasn’t generated it yet. 
> 
> The only recourse to this is to do a proc=only compile that does not include 
> the module-info.java class followed by a full compile with proc=none. Personally,
> I think this is a but in javac but I suspect it would be a major change to fix it.
> 
> I have meant to write a blog post on all of this basically to ask the question,
> “Is JPMS really worth it”?
> 
> Ralph
> 
>> On May 29, 2021, at 11:06 AM, Matt Sicker <bo...@gmail.com> wrote:
>> 
>> Hey all, that itch is back to figure out how to complete my previous
>> attempt at the plugin system updates (started a little over a year ago
>> and put on pause at some point). I'm currently working on getting that
>> branch back up to working again with master (mostly needing to update
>> module-info files), and I'm hoping to map out and figure out the path
>> forward on completing it.
>> 
>> With that being said, one major aspect of this project will be
>> explaining how the 2.x plugin system works, how it relates to the 3.x
>> system, and the gist of how I'd like to restructure things. Does
>> anyone here have any preferences on how I share some docs about this?
>> I could record some videos that go over code and such, or I could
>> write things up blog style. Note that these docs would be focused at
>> fellow log4j developers and not necessarily the user manual. I still
>> plan to have proper API docs and manual pages for the system, but
>> those would be targeted for users writing their own extensions mainly.
>> 
>> Link for background info:
>> https://issues.apache.org/jira/browse/LOG4J2-2803
>> 
> 
> 
> 

Re: [LOG4J2-2803] Getting back to the plugin system rewrite

[Matt Sicker <bo...@gmail.com>]

I'm having some fun right now figuring out the proper module openings
to enable for Java 16, so yes, I agree that the module system has been
a pain in the ass. I do like the idea of ensuring the output of the
annotation processor is a simple service loader class that could be
written by hand if needed, too (which could help simplify all the
usages in the wild of the packages attribute on configuration).

On Sat, 29 May 2021 at 19:40, Ralph Goers <ra...@dslextreme.com> wrote:
>
> We do need better docs on this. 3.0 is going to be a mixed bag. It will be easier
> for “normal” applications because plugins now use ServiceLoader instead of
> the Log4jPlugins.dat file. However, it is going to be more challenging for those
> developing plugins if they use the Java Platform Module System.
>
> As one might imagine, the compiler now has a modulePath in addition to a
> classPath. When no module-info.java is present Maven will set things up to
> only use the classPath, which will work as normal. However, when a module-info.java
> file is present than Maven will configure the project and its tests to use the
> modulePath. By itself this isn’t a problem. However, the Java compiler ONLY
> looks for Annotation processors automatically on the classPath. This means
> anyone using JPMS while compiling plugins will have to explicitly configure
> the annotation processor.
>
> There is a second issue when using JPMS. While our annotation processor
> generates the service class and service definition file, JPMS requires that the
> Log4jPlugins service class must be declared in module-info.java. This is
> somewhat of a problem because the compiler will fail if it finds the
> module-info.java references the Log4jPlugins service class because the
> annotation processor hasn’t generated it yet.
>
> The only recourse to this is to do a proc=only compile that does not include
> the module-info.java class followed by a full compile with proc=none. Personally,
> I think this is a but in javac but I suspect it would be a major change to fix it.
>
> I have meant to write a blog post on all of this basically to ask the question,
> “Is JPMS really worth it”?
>
> Ralph
>
> > On May 29, 2021, at 11:06 AM, Matt Sicker <bo...@gmail.com> wrote:
> >
> > Hey all, that itch is back to figure out how to complete my previous
> > attempt at the plugin system updates (started a little over a year ago
> > and put on pause at some point). I'm currently working on getting that
> > branch back up to working again with master (mostly needing to update
> > module-info files), and I'm hoping to map out and figure out the path
> > forward on completing it.
> >
> > With that being said, one major aspect of this project will be
> > explaining how the 2.x plugin system works, how it relates to the 3.x
> > system, and the gist of how I'd like to restructure things. Does
> > anyone here have any preferences on how I share some docs about this?
> > I could record some videos that go over code and such, or I could
> > write things up blog style. Note that these docs would be focused at
> > fellow log4j developers and not necessarily the user manual. I still
> > plan to have proper API docs and manual pages for the system, but
> > those would be targeted for users writing their own extensions mainly.
> >
> > Link for background info:
> > https://issues.apache.org/jira/browse/LOG4J2-2803
> >
>
>

Re: [LOG4J2-2803] Getting back to the plugin system rewrite

[Ralph Goers <ra...@dslextreme.com>]

We do need better docs on this. 3.0 is going to be a mixed bag. It will be easier 
for “normal” applications because plugins now use ServiceLoader instead of 
the Log4jPlugins.dat file. However, it is going to be more challenging for those 
developing plugins if they use the Java Platform Module System.

As one might imagine, the compiler now has a modulePath in addition to a 
classPath. When no module-info.java is present Maven will set things up to 
only use the classPath, which will work as normal. However, when a module-info.java 
file is present than Maven will configure the project and its tests to use the 
modulePath. By itself this isn’t a problem. However, the Java compiler ONLY 
looks for Annotation processors automatically on the classPath. This means 
anyone using JPMS while compiling plugins will have to explicitly configure 
the annotation processor. 

There is a second issue when using JPMS. While our annotation processor 
generates the service class and service definition file, JPMS requires that the 
Log4jPlugins service class must be declared in module-info.java. This is 
somewhat of a problem because the compiler will fail if it finds the 
module-info.java references the Log4jPlugins service class because the 
annotation processor hasn’t generated it yet. 

The only recourse to this is to do a proc=only compile that does not include 
the module-info.java class followed by a full compile with proc=none. Personally,
I think this is a but in javac but I suspect it would be a major change to fix it.

I have meant to write a blog post on all of this basically to ask the question,
“Is JPMS really worth it”?

Ralph

> On May 29, 2021, at 11:06 AM, Matt Sicker <bo...@gmail.com> wrote:
> 
> Hey all, that itch is back to figure out how to complete my previous
> attempt at the plugin system updates (started a little over a year ago
> and put on pause at some point). I'm currently working on getting that
> branch back up to working again with master (mostly needing to update
> module-info files), and I'm hoping to map out and figure out the path
> forward on completing it.
> 
> With that being said, one major aspect of this project will be
> explaining how the 2.x plugin system works, how it relates to the 3.x
> system, and the gist of how I'd like to restructure things. Does
> anyone here have any preferences on how I share some docs about this?
> I could record some videos that go over code and such, or I could
> write things up blog style. Note that these docs would be focused at
> fellow log4j developers and not necessarily the user manual. I still
> plan to have proper API docs and manual pages for the system, but
> those would be targeted for users writing their own extensions mainly.
> 
> Link for background info:
> https://issues.apache.org/jira/browse/LOG4J2-2803
>