You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by Volkan Yazıcı <vo...@gmail.com> on 2021/06/16 08:28:37 UTC
Bazel for builds
*[First and foremost, this is my personal Tweet and there I explicitly
stated that "*I* can offer" the amount for such a work. After completing
such a project, we need to have a consensus within us to merge it. I want
to explicitly avoid any unintended misunderstandings.]*
Gary, it was you who said "If the tooling is a problem, then let's look at
that first." I definitely had thought the same and evaluated Bazel myself
for quite some time a couple of months ago. Both Maven and Gradle have
quite some limitations for monorepos similar to Log4j. I agree that Bazel
is a total stranger to us (hence the maintenance problem), IDE support lags
behind significantly, etc. Yet, it is a tool designed for fast, correct,
and reproducible build of monorepos – a tool that is so-called to be a
perfect fit for us. My personal evaluation of Bazel did not yield a
positive outcome. Before crossing out this option, I wanted to ask around
to some Bazel experts for their feedback and that is what I did. In
summary, the outcome was again negative – which I was intending to discuss
in detail in the upcoming online meeting.
Long story short, rather than complaining, I gave it a spin myself. It
wasn't convincing enough, I got in touch with an expert, researched the
feasibility of such a goal and how much it would cost if I wanted it to be
done by somebody else.
If you have other ideas on how we can address "*the tooling problem"*, I am
all ears.
*P.S.* I am changing the subject of the thread, since I think it makes the
issue personal, which I believe wasn't your intention.
On Wed, Jun 16, 2021 at 9:55 AM Gary Gregory <ga...@gmail.com> wrote:
> Volkan Yazıcı (@yazicivo) tweeted at 3:34 PM on Sun, Jun 13, 2021:
> Do you know of a Bazel ninja who can migrate Log4j's Maven builds? I can
> offer €2k (and maybe more) for such a service.
> (https://twitter.com/yazicivo/status/1404160220025217027?s=03)
>
> How do we maintain something like this? Sounds like a terrible idea to me.
>
> Gary
>
Re: Bazel for builds
Posted by Ralph Goers <ra...@dslextreme.com>.
> On Jun 19, 2021, at 11:42 AM, Volkan Yazıcı <vo...@gmail.com> wrote:
>
> Why do we need both a -test and -unittest modules? Can't we configure
> surefire and failsafe to simply run the tests in the "main" of -test module?
a) The way Maven currently supports JPMS unit testing I can’t be sure that
placing the unit tests in a separate Maven module would work at all.
b) In a standard build “main” classes are built before test classes. But the
unit tests require classes from the test jar so those classes must be compiled
and packaged prior to running the tests.
Ralph
>
> A back of the envelope calculate yields the following:
>
> $ grep '<type>test-jar</type>' -RHn . -C 3 | grep '<artifactId>log4j' | sed
> -r 's/.*<artifactId>(.*)<\/artifactId>/\1/g' | sort | uniq
> log4j-api
> log4j-core
> log4j-jdbc
> log4j-layout-jackson
> log4j-layout-template-json
> log4j-plugins
>
> layout testJars are needed by perf. jdbc testJar is needed by jdbc-dbcp2.
> We can work these both out and avoid them. There remains only 3 modules
> that we need to create a -test module for: api, core, and plugins.
>
>
>
> On Sat, Jun 19, 2021 at 7:33 PM Ralph Goers <ra...@dslextreme.com>
> wrote:
>
>> If you did that then it would require that instead of the Log4j-api module
>> producing the log4j-api jar and log4j-api test jar you would end up with
>>
>> * log4j-api - with no tests.
>> * log4j-api-test - creates the test jar.
>> * log4j-api-unittest - contains and runs the unit tests.
>>
>> I’d rather live with the single module weirdness than need 3 modules.
>>
>> Ralph
>>
>>> On Jun 19, 2021, at 9:53 AM, Matt Sicker <bo...@gmail.com> wrote:
>>>
>>> I think the idea would be to separate the reusable test code into their
>> own
>>> main modules which depend on JUnit and such which are only consumed by
>> test
>>> modules. I think this approach could potentially work, but we’d end up
>> with
>>> a few more published jars.
>>>
>>> On Sat, Jun 19, 2021 at 11:28 Ralph Goers <ra...@dslextreme.com>
>>> wrote:
>>>
>>>>
>>>>
>>>>> On Jun 19, 2021, at 3:45 AM, Volkan Yazıcı <vo...@gmail.com>
>>>> wrote:
>>>>>
>>>>> That README is such a gem Ralph! Thanks for documenting all that
>>>> suffering.
>>>>> (README says Log4j 2 supports JPMS, shouldn't it rather be Log4j 3?)
>>>>
>>>> No, I consider it Log4j 2 version 3.0. IOW, the 2 is just part of the
>>>> name. I can see
>>>> where you might prefer just Log4j 3.0 though.
>>>>
>>>>>
>>>>> There are actually pretty few modules whose test JARs are used by the
>>>>> downstream, mostly by log4j-perf. Could it be a viable option to move
>> the
>>>>> benchmarks in log4j-perf to their associated modules (under src/perf?)
>>>>> and/or convert tests that produce test JARs into separate Maven
>> modules,
>>>>> e.g., log4j-core-test, log4j-layout-template-json-test? With my pretty
>>>>> limited understanding of JPMS, I have the impression that this might
>> ease
>>>>> the build pain to a certain extent.
>>>>
>>>> I don’t think that is quite true. The test classes included in the test
>>>> jars of
>>>> log4j-api, log4j-plugins, and log4j-core are all used by the unit tests
>> in
>>>> the
>>>> other modules. Log4j-perf doesn’t even have any test classes and doesn’t
>>>> have any dependencies on test jars.
>>>>
>>>> Ralph
>>>>
>>
>>
>>
Re: Bazel for builds
Posted by Volkan Yazıcı <vo...@gmail.com>.
Why do we need both a -test and -unittest modules? Can't we configure
surefire and failsafe to simply run the tests in the "main" of -test module?
A back of the envelope calculate yields the following:
$ grep '<type>test-jar</type>' -RHn . -C 3 | grep '<artifactId>log4j' | sed
-r 's/.*<artifactId>(.*)<\/artifactId>/\1/g' | sort | uniq
log4j-api
log4j-core
log4j-jdbc
log4j-layout-jackson
log4j-layout-template-json
log4j-plugins
layout testJars are needed by perf. jdbc testJar is needed by jdbc-dbcp2.
We can work these both out and avoid them. There remains only 3 modules
that we need to create a -test module for: api, core, and plugins.
On Sat, Jun 19, 2021 at 7:33 PM Ralph Goers <ra...@dslextreme.com>
wrote:
> If you did that then it would require that instead of the Log4j-api module
> producing the log4j-api jar and log4j-api test jar you would end up with
>
> * log4j-api - with no tests.
> * log4j-api-test - creates the test jar.
> * log4j-api-unittest - contains and runs the unit tests.
>
> I’d rather live with the single module weirdness than need 3 modules.
>
> Ralph
>
> > On Jun 19, 2021, at 9:53 AM, Matt Sicker <bo...@gmail.com> wrote:
> >
> > I think the idea would be to separate the reusable test code into their
> own
> > main modules which depend on JUnit and such which are only consumed by
> test
> > modules. I think this approach could potentially work, but we’d end up
> with
> > a few more published jars.
> >
> > On Sat, Jun 19, 2021 at 11:28 Ralph Goers <ra...@dslextreme.com>
> > wrote:
> >
> >>
> >>
> >>> On Jun 19, 2021, at 3:45 AM, Volkan Yazıcı <vo...@gmail.com>
> >> wrote:
> >>>
> >>> That README is such a gem Ralph! Thanks for documenting all that
> >> suffering.
> >>> (README says Log4j 2 supports JPMS, shouldn't it rather be Log4j 3?)
> >>
> >> No, I consider it Log4j 2 version 3.0. IOW, the 2 is just part of the
> >> name. I can see
> >> where you might prefer just Log4j 3.0 though.
> >>
> >>>
> >>> There are actually pretty few modules whose test JARs are used by the
> >>> downstream, mostly by log4j-perf. Could it be a viable option to move
> the
> >>> benchmarks in log4j-perf to their associated modules (under src/perf?)
> >>> and/or convert tests that produce test JARs into separate Maven
> modules,
> >>> e.g., log4j-core-test, log4j-layout-template-json-test? With my pretty
> >>> limited understanding of JPMS, I have the impression that this might
> ease
> >>> the build pain to a certain extent.
> >>
> >> I don’t think that is quite true. The test classes included in the test
> >> jars of
> >> log4j-api, log4j-plugins, and log4j-core are all used by the unit tests
> in
> >> the
> >> other modules. Log4j-perf doesn’t even have any test classes and doesn’t
> >> have any dependencies on test jars.
> >>
> >> Ralph
> >>
>
>
>
Re: Bazel for builds
Posted by Ralph Goers <ra...@dslextreme.com>.
If you did that then it would require that instead of the Log4j-api module
producing the log4j-api jar and log4j-api test jar you would end up with
* log4j-api - with no tests.
* log4j-api-test - creates the test jar.
* log4j-api-unittest - contains and runs the unit tests.
I’d rather live with the single module weirdness than need 3 modules.
Ralph
> On Jun 19, 2021, at 9:53 AM, Matt Sicker <bo...@gmail.com> wrote:
>
> I think the idea would be to separate the reusable test code into their own
> main modules which depend on JUnit and such which are only consumed by test
> modules. I think this approach could potentially work, but we’d end up with
> a few more published jars.
>
> On Sat, Jun 19, 2021 at 11:28 Ralph Goers <ra...@dslextreme.com>
> wrote:
>
>>
>>
>>> On Jun 19, 2021, at 3:45 AM, Volkan Yazıcı <vo...@gmail.com>
>> wrote:
>>>
>>> That README is such a gem Ralph! Thanks for documenting all that
>> suffering.
>>> (README says Log4j 2 supports JPMS, shouldn't it rather be Log4j 3?)
>>
>> No, I consider it Log4j 2 version 3.0. IOW, the 2 is just part of the
>> name. I can see
>> where you might prefer just Log4j 3.0 though.
>>
>>>
>>> There are actually pretty few modules whose test JARs are used by the
>>> downstream, mostly by log4j-perf. Could it be a viable option to move the
>>> benchmarks in log4j-perf to their associated modules (under src/perf?)
>>> and/or convert tests that produce test JARs into separate Maven modules,
>>> e.g., log4j-core-test, log4j-layout-template-json-test? With my pretty
>>> limited understanding of JPMS, I have the impression that this might ease
>>> the build pain to a certain extent.
>>
>> I don’t think that is quite true. The test classes included in the test
>> jars of
>> log4j-api, log4j-plugins, and log4j-core are all used by the unit tests in
>> the
>> other modules. Log4j-perf doesn’t even have any test classes and doesn’t
>> have any dependencies on test jars.
>>
>> Ralph
>>
Re: Bazel for builds
Posted by Matt Sicker <bo...@gmail.com>.
I think the idea would be to separate the reusable test code into their own
main modules which depend on JUnit and such which are only consumed by test
modules. I think this approach could potentially work, but we’d end up with
a few more published jars.
On Sat, Jun 19, 2021 at 11:28 Ralph Goers <ra...@dslextreme.com>
wrote:
>
>
> > On Jun 19, 2021, at 3:45 AM, Volkan Yazıcı <vo...@gmail.com>
> wrote:
> >
> > That README is such a gem Ralph! Thanks for documenting all that
> suffering.
> > (README says Log4j 2 supports JPMS, shouldn't it rather be Log4j 3?)
>
> No, I consider it Log4j 2 version 3.0. IOW, the 2 is just part of the
> name. I can see
> where you might prefer just Log4j 3.0 though.
>
> >
> > There are actually pretty few modules whose test JARs are used by the
> > downstream, mostly by log4j-perf. Could it be a viable option to move the
> > benchmarks in log4j-perf to their associated modules (under src/perf?)
> > and/or convert tests that produce test JARs into separate Maven modules,
> > e.g., log4j-core-test, log4j-layout-template-json-test? With my pretty
> > limited understanding of JPMS, I have the impression that this might ease
> > the build pain to a certain extent.
>
> I don’t think that is quite true. The test classes included in the test
> jars of
> log4j-api, log4j-plugins, and log4j-core are all used by the unit tests in
> the
> other modules. Log4j-perf doesn’t even have any test classes and doesn’t
> have any dependencies on test jars.
>
> Ralph
>
Re: Bazel for builds
Posted by Ralph Goers <ra...@dslextreme.com>.
> On Jun 19, 2021, at 3:45 AM, Volkan Yazıcı <vo...@gmail.com> wrote:
>
> That README is such a gem Ralph! Thanks for documenting all that suffering.
> (README says Log4j 2 supports JPMS, shouldn't it rather be Log4j 3?)
No, I consider it Log4j 2 version 3.0. IOW, the 2 is just part of the name. I can see
where you might prefer just Log4j 3.0 though.
>
> There are actually pretty few modules whose test JARs are used by the
> downstream, mostly by log4j-perf. Could it be a viable option to move the
> benchmarks in log4j-perf to their associated modules (under src/perf?)
> and/or convert tests that produce test JARs into separate Maven modules,
> e.g., log4j-core-test, log4j-layout-template-json-test? With my pretty
> limited understanding of JPMS, I have the impression that this might ease
> the build pain to a certain extent.
I don’t think that is quite true. The test classes included in the test jars of
log4j-api, log4j-plugins, and log4j-core are all used by the unit tests in the
other modules. Log4j-perf doesn’t even have any test classes and doesn’t
have any dependencies on test jars.
Ralph
Re: Bazel for builds
Posted by Volkan Yazıcı <vo...@gmail.com>.
That README is such a gem Ralph! Thanks for documenting all that suffering.
(README says Log4j 2 supports JPMS, shouldn't it rather be Log4j 3?)
There are actually pretty few modules whose test JARs are used by the
downstream, mostly by log4j-perf. Could it be a viable option to move the
benchmarks in log4j-perf to their associated modules (under src/perf?)
and/or convert tests that produce test JARs into separate Maven modules,
e.g., log4j-core-test, log4j-layout-template-json-test? With my pretty
limited understanding of JPMS, I have the impression that this might ease
the build pain to a certain extent.
On Sat, Jun 19, 2021 at 12:53 AM Ralph Goers <ra...@dslextreme.com>
wrote:
>
>
> > On Jun 18, 2021, at 2:19 PM, Jochen Wiedmann <jo...@gmail.com>
> wrote:
> >
> > On Fri, Jun 18, 2021 at 6:20 PM Ralph Goers <ra...@dslextreme.com>
> wrote:
> >
> >> 2. The configuration does “strange things” because Maven doesn’t
> support creating a
> >> JPMS module, JPMS test module, and running unit tests in a single Maven
> module
> >> out of the box while also avoiding
> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8265826.
> >> The “strange configuration” makes that work. Log4j-Plugins is even more
> “strange”
> >> because in addition to the above it also has to compile the annotation
> processor
> >> without the module-info.java prior to doing the “build compile” so that
> the annotation
> >> processor exists.
> >
> > Are there existing Maven issues, that I could have a look at? Or, is
> > the problem reproducible with a relatively simple example?
>
> I haven’t created any Maven issues for this because Maven is working as
> designed, for the
> most part. I did create
> https://issues.apache.org/jira/browse/MCOMPILER-461 but that is
> for a different problem I encountered.
>
> For the JPMS compiler bug I created
> https://github.com/rgoers/jpms-compile-fails. That
> bug was fixed in Java 17 so if we upgrade I could fix some of the
> “strangeness”. I have not
> tested with Java 17 to see if that also somehow fixes MCOMPILER-461.
>
> But to create a sample project just use the standard Maven Archetype to
> create a project.
> Then convert it to JPMS modules - you will need a module-info.java for
> both the main
> source and for the unit tests but the unit tests will “open” the main
> module and all the test
> classes will be in the same package space as the main source as is typical
> for normal testing.
>
> Next create some classes that are used both in your unit tests and will be
> used in downstream
> tests. This is where it gets ugly. These tests have to be packaged in
> their own jar and cannot
> use the same package space as the main source, so typically you just would
> add .test to
> whatever the main source package is and put all these classes under that.
> You will find that
> you need to compile these separately from the other test classes and use a
> module-info.java
> that names the module properly. Once you have compiled these you would
> then package them
> as the test jar for the project. Once you do that you have to delete the
> module-info.class file that
> was generated and then compile the rest of the test classes with their
> module-info.java file.
>
> As you can see, the complication is that JPMS doesn’t let you just package
> all your test crap in
> a jar and send it downstream for other modules to use as is typically done
> without it. If you do
> that you will get an error because it duplicates the main source modules
> packages.
>
> Unfortunately, the test jar can’t be built in a separate module from the
> main source because
> it has dependencies on the main source and the test classes use the
> classes in the test jar.
>
> So the problem really is that JPMS has made it so that any time you need
> to package a test
> jar you are going to have to do all this crap. Maven simply doesn’t have a
> convention to
> handle this. The fact that I also encountered a bug in the compiler just
> made things worse
> as that required that the module-info.java be compiled after all the
> classes were compiled.
>
> https://github.com/apache/logging-log4j2/blob/master/log4j-core/README.md
> pretty much
> documents why I had to resort to all the messiness.
>
> Ralph
>
>
>
Re: Bazel for builds
Posted by Ralph Goers <ra...@dslextreme.com>.
> On Jun 18, 2021, at 2:19 PM, Jochen Wiedmann <jo...@gmail.com> wrote:
>
> On Fri, Jun 18, 2021 at 6:20 PM Ralph Goers <ra...@dslextreme.com> wrote:
>
>> 2. The configuration does “strange things” because Maven doesn’t support creating a
>> JPMS module, JPMS test module, and running unit tests in a single Maven module
>> out of the box while also avoiding https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8265826.
>> The “strange configuration” makes that work. Log4j-Plugins is even more “strange”
>> because in addition to the above it also has to compile the annotation processor
>> without the module-info.java prior to doing the “build compile” so that the annotation
>> processor exists.
>
> Are there existing Maven issues, that I could have a look at? Or, is
> the problem reproducible with a relatively simple example?
I haven’t created any Maven issues for this because Maven is working as designed, for the
most part. I did create https://issues.apache.org/jira/browse/MCOMPILER-461 but that is
for a different problem I encountered.
For the JPMS compiler bug I created https://github.com/rgoers/jpms-compile-fails. That
bug was fixed in Java 17 so if we upgrade I could fix some of the “strangeness”. I have not
tested with Java 17 to see if that also somehow fixes MCOMPILER-461.
But to create a sample project just use the standard Maven Archetype to create a project.
Then convert it to JPMS modules - you will need a module-info.java for both the main
source and for the unit tests but the unit tests will “open” the main module and all the test
classes will be in the same package space as the main source as is typical for normal testing.
Next create some classes that are used both in your unit tests and will be used in downstream
tests. This is where it gets ugly. These tests have to be packaged in their own jar and cannot
use the same package space as the main source, so typically you just would add .test to
whatever the main source package is and put all these classes under that. You will find that
you need to compile these separately from the other test classes and use a module-info.java
that names the module properly. Once you have compiled these you would then package them
as the test jar for the project. Once you do that you have to delete the module-info.class file that
was generated and then compile the rest of the test classes with their module-info.java file.
As you can see, the complication is that JPMS doesn’t let you just package all your test crap in
a jar and send it downstream for other modules to use as is typically done without it. If you do
that you will get an error because it duplicates the main source modules packages.
Unfortunately, the test jar can’t be built in a separate module from the main source because
it has dependencies on the main source and the test classes use the classes in the test jar.
So the problem really is that JPMS has made it so that any time you need to package a test
jar you are going to have to do all this crap. Maven simply doesn’t have a convention to
handle this. The fact that I also encountered a bug in the compiler just made things worse
as that required that the module-info.java be compiled after all the classes were compiled.
https://github.com/apache/logging-log4j2/blob/master/log4j-core/README.md pretty much
documents why I had to resort to all the messiness.
Ralph
Re: Bazel for builds
Posted by Jochen Wiedmann <jo...@gmail.com>.
On Fri, Jun 18, 2021 at 6:20 PM Ralph Goers <ra...@dslextreme.com> wrote:
> 2. The configuration does “strange things” because Maven doesn’t support creating a
> JPMS module, JPMS test module, and running unit tests in a single Maven module
> out of the box while also avoiding https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8265826.
> The “strange configuration” makes that work. Log4j-Plugins is even more “strange”
> because in addition to the above it also has to compile the annotation processor
> without the module-info.java prior to doing the “build compile” so that the annotation
> processor exists.
Are there existing Maven issues, that I could have a look at? Or, is
the problem reproducible with a relatively simple example?
Jochen
--
Look, that's why there's rules, understand? So that you think before
you break 'em.
-- (Terry Pratchett, Thief of Time)
Re: Bazel for builds
Posted by Ralph Goers <ra...@dslextreme.com>.
> On Jun 18, 2021, at 8:05 AM, Jochen Wiedmann <jo...@gmail.com> wrote:
>
> On Wed, Jun 16, 2021 at 1:31 PM Volkan Yazıcı <vo...@gmail.com> wrote:
>
>> In essence, the build is taking more than it should. Maven "verify" is
>> taking more than half an hour and "site" is taking ages. This in addition
>> to impeding the development cycle, cripples the release process too.
>
> And why do you blame that on the build tool, rather than the
> configuration, which seems to do rather strange things. Maven's
> performance is quite decent, if given a chance. Okay, "site" may be an
> exception, but you should be able to replace Doxia with another site
> generation too,, if you are ready to drop all those nice reports, that
> Maven generates, and that users like me find very useful.
1. I am not ready to drop Maven.
2. The configuration does “strange things” because Maven doesn’t support creating a
JPMS module, JPMS test module, and running unit tests in a single Maven module
out of the box while also avoiding https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8265826.
The “strange configuration” makes that work. Log4j-Plugins is even more “strange”
because in addition to the above it also has to compile the annotation processor
without the module-info.java prior to doing the “build compile” so that the annotation
processor exists.
3. I am of the opinion that we should move most of the log4j site from
https://github.com/apache/logging-log4j2 to https://github.com/apache/logging-log4j-site/tree/asf-staging.
Ralph
Re: Bazel for builds
Posted by Jochen Wiedmann <jo...@gmail.com>.
On Wed, Jun 16, 2021 at 1:31 PM Volkan Yazıcı <vo...@gmail.com> wrote:
> In essence, the build is taking more than it should. Maven "verify" is
> taking more than half an hour and "site" is taking ages. This in addition
> to impeding the development cycle, cripples the release process too.
And why do you blame that on the build tool, rather than the
configuration, which seems to do rather strange things. Maven's
performance is quite decent, if given a chance. Okay, "site" may be an
exception, but you should be able to replace Doxia with another site
generation too,, if you are ready to drop all those nice reports, that
Maven generates, and that users like me find very useful.
Jochen
--
Look, that's why there's rules, understand? So that you think before
you break 'em.
-- (Terry Pratchett, Thief of Time)
Re: Bazel for builds
Posted by Jeanderson <je...@gmail.com>.
I know that Gradle supports Build cache locally and remotely (see [1]).
This can be useful to speed-up the process.
Gradle also provides parallel build (there are other features related to
performance here [2])
I'm unable to provide some concrete thoughts on how that may work for Log4J
because I'm not familiar with how Log4J is structured.
However, I can take a look at how the Maven build looks now and iterate on
top of that.
[1] https://docs.gradle.org/current/userguide/build_cache.html
[2] https://docs.gradle.org/current/userguide/performance.html
Em qui., 17 de jun. de 2021 às 09:14, Volkan Yazıcı <vo...@gmail.com>
escreveu:
> Regarding distributed compilation... We can get in touch with ASF to have
> some VMs for a build farm. That is, we will still run the build locally,
> though the compilation will be delegated to a cluster of nodes, which in
> return will speed up the process notably. Back in my graduate days, I was a
> big fan of distcc. I often used the build farm to test my patches against
> PostgreSQL – yes, back then I was contributing to PostgreSQL, the good old
> days.
>
> [Note that I am still negative with regards to Bazel. That said, I am open
> to discussion.]
>
> On Wed, Jun 16, 2021 at 5:50 PM Ralph Goers <ra...@dslextreme.com>
> wrote:
>
> >
> >
> > > On Jun 16, 2021, at 8:29 AM, Matt Sicker <bo...@gmail.com> wrote:
> > >
> > > I should note some possible benefits we could get from Bazel in theory:
> > >
> > > * Ability to offload builds/tests/etc. to remote build clusters. If
> > > you've ever used things like distcc for compiling things across
> > > multiple machines (handy for compiling C or C++ binaries for example),
> > > it sounds similar.
> >
> > I don’t get this. When I build locally I only have one machine. With CI
> we
> > run the build on as many environments as we want.
> >
> >
> > > * Ability to combine multiple languages together. If we wanted to,
> > > we'd be able to combine the log4net, log4cxx, and any other future
> > > subprojects into a single one. This would really only be useful if we
> > > had a unified configuration format or similar.
> >
> > OK, but I would never envision doing this.
> >
> > > * More granular builds (should be able to recompile only parts that
> > changed)
> >
> > Maven is supposed to do this. I’ve often wondered why it never works.
> >
> >
> > >
> > > There are some disadvantages, though:
> > >
> > > * No good IDE support yet AFAIK.
> > > * Build files are somewhat tedious compared to Maven's convention over
> > > configuration style.
> > > * Can be over complicated for a project that only uses a single
> > > programming language or compiler (possibly useful when you need
> > > multiple versions of Java, but we already have the Maven toolchains
> > > feature for that).
> > > * Has a bit of a bootstrapping problem: Bazel seems like it works best
> > > when everyone around you is also using it. Being able to reuse
> > > configurations and such is handy. This applies to pretty much any
> > > build system.
> > >
> > > As for the Maven modularity (below), this might be something that can
> > > be addressed during the plugin system updates (potentially making
> > > log4j-plugins purely the annotation processor).
> >
> > The above isn’t really possible. The annotation processor build, the
> > annotation
> > processor runtime, and the plugin system during execution pretty much
> > share
> > everything in Log4j-plugins. I believe when it was split from core the
> > smallest
> > set of stuff that had to be together was broken out.
> >
> >
> > Ralph
> >
> >
> >
>
Re: Bazel for builds
Posted by Volkan Yazıcı <vo...@gmail.com>.
Regarding distributed compilation... We can get in touch with ASF to have
some VMs for a build farm. That is, we will still run the build locally,
though the compilation will be delegated to a cluster of nodes, which in
return will speed up the process notably. Back in my graduate days, I was a
big fan of distcc. I often used the build farm to test my patches against
PostgreSQL – yes, back then I was contributing to PostgreSQL, the good old
days.
[Note that I am still negative with regards to Bazel. That said, I am open
to discussion.]
On Wed, Jun 16, 2021 at 5:50 PM Ralph Goers <ra...@dslextreme.com>
wrote:
>
>
> > On Jun 16, 2021, at 8:29 AM, Matt Sicker <bo...@gmail.com> wrote:
> >
> > I should note some possible benefits we could get from Bazel in theory:
> >
> > * Ability to offload builds/tests/etc. to remote build clusters. If
> > you've ever used things like distcc for compiling things across
> > multiple machines (handy for compiling C or C++ binaries for example),
> > it sounds similar.
>
> I don’t get this. When I build locally I only have one machine. With CI we
> run the build on as many environments as we want.
>
>
> > * Ability to combine multiple languages together. If we wanted to,
> > we'd be able to combine the log4net, log4cxx, and any other future
> > subprojects into a single one. This would really only be useful if we
> > had a unified configuration format or similar.
>
> OK, but I would never envision doing this.
>
> > * More granular builds (should be able to recompile only parts that
> changed)
>
> Maven is supposed to do this. I’ve often wondered why it never works.
>
>
> >
> > There are some disadvantages, though:
> >
> > * No good IDE support yet AFAIK.
> > * Build files are somewhat tedious compared to Maven's convention over
> > configuration style.
> > * Can be over complicated for a project that only uses a single
> > programming language or compiler (possibly useful when you need
> > multiple versions of Java, but we already have the Maven toolchains
> > feature for that).
> > * Has a bit of a bootstrapping problem: Bazel seems like it works best
> > when everyone around you is also using it. Being able to reuse
> > configurations and such is handy. This applies to pretty much any
> > build system.
> >
> > As for the Maven modularity (below), this might be something that can
> > be addressed during the plugin system updates (potentially making
> > log4j-plugins purely the annotation processor).
>
> The above isn’t really possible. The annotation processor build, the
> annotation
> processor runtime, and the plugin system during execution pretty much
> share
> everything in Log4j-plugins. I believe when it was split from core the
> smallest
> set of stuff that had to be together was broken out.
>
>
> Ralph
>
>
>
Re: Bazel for builds
Posted by Ralph Goers <ra...@dslextreme.com>.
> On Jun 16, 2021, at 8:29 AM, Matt Sicker <bo...@gmail.com> wrote:
>
> I should note some possible benefits we could get from Bazel in theory:
>
> * Ability to offload builds/tests/etc. to remote build clusters. If
> you've ever used things like distcc for compiling things across
> multiple machines (handy for compiling C or C++ binaries for example),
> it sounds similar.
I don’t get this. When I build locally I only have one machine. With CI we
run the build on as many environments as we want.
> * Ability to combine multiple languages together. If we wanted to,
> we'd be able to combine the log4net, log4cxx, and any other future
> subprojects into a single one. This would really only be useful if we
> had a unified configuration format or similar.
OK, but I would never envision doing this.
> * More granular builds (should be able to recompile only parts that changed)
Maven is supposed to do this. I’ve often wondered why it never works.
>
> There are some disadvantages, though:
>
> * No good IDE support yet AFAIK.
> * Build files are somewhat tedious compared to Maven's convention over
> configuration style.
> * Can be over complicated for a project that only uses a single
> programming language or compiler (possibly useful when you need
> multiple versions of Java, but we already have the Maven toolchains
> feature for that).
> * Has a bit of a bootstrapping problem: Bazel seems like it works best
> when everyone around you is also using it. Being able to reuse
> configurations and such is handy. This applies to pretty much any
> build system.
>
> As for the Maven modularity (below), this might be something that can
> be addressed during the plugin system updates (potentially making
> log4j-plugins purely the annotation processor).
The above isn’t really possible. The annotation processor build, the annotation
processor runtime, and the plugin system during execution pretty much share
everything in Log4j-plugins. I believe when it was split from core the smallest
set of stuff that had to be together was broken out.
Ralph
Re: Bazel for builds
Posted by Matt Sicker <bo...@gmail.com>.
I should note some possible benefits we could get from Bazel in theory:
* Ability to offload builds/tests/etc. to remote build clusters. If
you've ever used things like distcc for compiling things across
multiple machines (handy for compiling C or C++ binaries for example),
it sounds similar.
* Ability to combine multiple languages together. If we wanted to,
we'd be able to combine the log4net, log4cxx, and any other future
subprojects into a single one. This would really only be useful if we
had a unified configuration format or similar.
* More granular builds (should be able to recompile only parts that changed)
There are some disadvantages, though:
* No good IDE support yet AFAIK.
* Build files are somewhat tedious compared to Maven's convention over
configuration style.
* Can be over complicated for a project that only uses a single
programming language or compiler (possibly useful when you need
multiple versions of Java, but we already have the Maven toolchains
feature for that).
* Has a bit of a bootstrapping problem: Bazel seems like it works best
when everyone around you is also using it. Being able to reuse
configurations and such is handy. This applies to pretty much any
build system.
As for the Maven modularity (below), this might be something that can
be addressed during the plugin system updates (potentially making
log4j-plugins purely the annotation processor).
On Wed, Jun 16, 2021 at 9:09 AM Matt Sicker <bo...@gmail.com> wrote:
>
> I’ve noticed that attempting to support Java modules properly with Maven requires a lot of non standard configuration. I don’t know if anything can help there.
>
> On Wed, Jun 16, 2021 at 07:11 Jeanderson <je...@gmail.com> wrote:
>>
>> Thanks for the input Volkan!
>>
>>
>> > By switching to another build tool (Bazel or anything else), I expect to
>> > speed up these processes, maybe, by having more fine-grained control on the
>> > dependency graph and avoiding unnecessary steps.
>> >
>>
>> I see quite often people switching from Maven to Gradle because Gradle
>> gives some flexibility on customizing the build cycle
>> and also because some people find it more convenient to use a scripting
>> language (rather than XML) to define the build process.
>> It seems to be a positive point in favor of Gradle when it comes to fine
>> tuning the build.
>>
>> Still, I'm curious about the use of Gradle in a monorepo.
>> Is there some particular reason to think that Gradle is not a good fit for
>> Log4J?
>>
>> About keeping Maven, wouldn't it be an alternative to use Build Profiles
>> [1] to skip unnecessary tasks?
>>
>> If others want to share some thoughts on the topic, it would be good too.
>> Maybe, that's something that I can do some investigation and contribute to
>> the community.
>>
>> [1]
>> https://maven.apache.org/guides/introduction/introduction-to-profiles.html
>>
>> Best,
>> Jeanderson Candido
Re: Bazel for builds
Posted by Ralph Goers <ra...@dslextreme.com>.
Actually, Maven has done a decent job of supporting modules for “standard” projects.
Our issue is that we are creating a test jars in log4j-api, log4j-core, and log4j-plugins.
These also need to comply with JPMS which means they can’t use the same package
space as the module they are part of.
In addition, JPMS has impacts on the way the annotation process works. We always
needed a separate compile step to first compile the annotation processor and then
compile everything while using it. JPMS complicates this since a module-info.java
cannot be present when compiling the annotation processor due to the compiler bug
I encountered. Annotation processing is also complicated for other modules since it
checks for a module-info.java even when proc=only is specified.
FWIW, the compiler bug I encountered has been fixed in Java 17. I don’t know if
they will back port it to any earlier releases. I am quite certain I could simplify the
build if we require the Java 17 compiler be used.
In short, Maven isn’t the cause of why log4j-api, log4j-core, and log4j-plugins are
so complicated. It is almost entirely due to JPMS and our use of annotation processing.
Ralph
> On Jun 16, 2021, at 7:09 AM, Matt Sicker <bo...@gmail.com> wrote:
>
> I’ve noticed that attempting to support Java modules properly with Maven
> requires a lot of non standard configuration. I don’t know if anything can
> help there.
>
> On Wed, Jun 16, 2021 at 07:11 Jeanderson <je...@gmail.com> wrote:
>
>> Thanks for the input Volkan!
>>
>>
>>> By switching to another build tool (Bazel or anything else), I expect to
>>> speed up these processes, maybe, by having more fine-grained control on
>> the
>>> dependency graph and avoiding unnecessary steps.
>>>
>>
>> I see quite often people switching from Maven to Gradle because Gradle
>> gives some flexibility on customizing the build cycle
>> and also because some people find it more convenient to use a scripting
>> language (rather than XML) to define the build process.
>> It seems to be a positive point in favor of Gradle when it comes to fine
>> tuning the build.
>>
>> Still, I'm curious about the use of Gradle in a monorepo.
>> Is there some particular reason to think that Gradle is not a good fit for
>> Log4J?
>>
>> About keeping Maven, wouldn't it be an alternative to use Build Profiles
>> [1] to skip unnecessary tasks?
>>
>> If others want to share some thoughts on the topic, it would be good too.
>> Maybe, that's something that I can do some investigation and contribute to
>> the community.
>>
>> [1]
>> https://maven.apache.org/guides/introduction/introduction-to-profiles.html
>>
>> Best,
>> Jeanderson Candido
>>
Re: Bazel for builds
Posted by Matt Sicker <bo...@gmail.com>.
I’ve noticed that attempting to support Java modules properly with Maven
requires a lot of non standard configuration. I don’t know if anything can
help there.
On Wed, Jun 16, 2021 at 07:11 Jeanderson <je...@gmail.com> wrote:
> Thanks for the input Volkan!
>
>
> > By switching to another build tool (Bazel or anything else), I expect to
> > speed up these processes, maybe, by having more fine-grained control on
> the
> > dependency graph and avoiding unnecessary steps.
> >
>
> I see quite often people switching from Maven to Gradle because Gradle
> gives some flexibility on customizing the build cycle
> and also because some people find it more convenient to use a scripting
> language (rather than XML) to define the build process.
> It seems to be a positive point in favor of Gradle when it comes to fine
> tuning the build.
>
> Still, I'm curious about the use of Gradle in a monorepo.
> Is there some particular reason to think that Gradle is not a good fit for
> Log4J?
>
> About keeping Maven, wouldn't it be an alternative to use Build Profiles
> [1] to skip unnecessary tasks?
>
> If others want to share some thoughts on the topic, it would be good too.
> Maybe, that's something that I can do some investigation and contribute to
> the community.
>
> [1]
> https://maven.apache.org/guides/introduction/introduction-to-profiles.html
>
> Best,
> Jeanderson Candido
>
Re: Bazel for builds
Posted by Jeanderson <je...@gmail.com>.
Thanks for the input Volkan!
> By switching to another build tool (Bazel or anything else), I expect to
> speed up these processes, maybe, by having more fine-grained control on the
> dependency graph and avoiding unnecessary steps.
>
I see quite often people switching from Maven to Gradle because Gradle
gives some flexibility on customizing the build cycle
and also because some people find it more convenient to use a scripting
language (rather than XML) to define the build process.
It seems to be a positive point in favor of Gradle when it comes to fine
tuning the build.
Still, I'm curious about the use of Gradle in a monorepo.
Is there some particular reason to think that Gradle is not a good fit for
Log4J?
About keeping Maven, wouldn't it be an alternative to use Build Profiles
[1] to skip unnecessary tasks?
If others want to share some thoughts on the topic, it would be good too.
Maybe, that's something that I can do some investigation and contribute to
the community.
[1]
https://maven.apache.org/guides/introduction/introduction-to-profiles.html
Best,
Jeanderson Candido
Re: Bazel for builds
Posted by Volkan Yazıcı <vo...@gmail.com>.
In essence, the build is taking more than it should. Maven "verify" is
taking more than half an hour and "site" is taking ages. This in addition
to impeding the development cycle, cripples the release process too.
By switching to another build tool (Bazel or anything else), I expect to
speed up these processes, maybe, by having more fine-grained control on the
dependency graph and avoiding unnecessary steps.
On Wed, Jun 16, 2021 at 12:26 PM Jeanderson <je...@gmail.com> wrote:
> Hi all,
>
> I'm not particularly involved with Log4J development but I've been
> following this list for a few months now just because I'm curious about
> Log4J development (and also because I'm also doing some R&D related to
> logging). In fact, this is my first ever post here.
>
> May I ask some clarifying questions?
> - What are the ongoing struggles that make Maven not a good fit for Log4J?
> - What are you expecting to achieve by switching to Bazel?
>
> I'm genuinely interested in this discussion and I hope that it may be
> beneficial here.
>
> Apologies in advance if this discussion is not convenient for this thread.
> In case you are wondering, no, I have no connections with Maven or Bazel.
>
> Best,
> Jeanderson Candido
>
>
>
> Em qua., 16 de jun. de 2021 às 10:28, Volkan Yazıcı <
> volkan.yazici@gmail.com>
> escreveu:
>
> > *[First and foremost, this is my personal Tweet and there I explicitly
> > stated that "*I* can offer" the amount for such a work. After completing
> > such a project, we need to have a consensus within us to merge it. I want
> > to explicitly avoid any unintended misunderstandings.]*
> >
> > Gary, it was you who said "If the tooling is a problem, then let's look
> at
> > that first." I definitely had thought the same and evaluated Bazel myself
> > for quite some time a couple of months ago. Both Maven and Gradle have
> > quite some limitations for monorepos similar to Log4j. I agree that Bazel
> > is a total stranger to us (hence the maintenance problem), IDE support
> lags
> > behind significantly, etc. Yet, it is a tool designed for fast, correct,
> > and reproducible build of monorepos – a tool that is so-called to be a
> > perfect fit for us. My personal evaluation of Bazel did not yield a
> > positive outcome. Before crossing out this option, I wanted to ask around
> > to some Bazel experts for their feedback and that is what I did. In
> > summary, the outcome was again negative – which I was intending to
> discuss
> > in detail in the upcoming online meeting.
> >
> > Long story short, rather than complaining, I gave it a spin myself. It
> > wasn't convincing enough, I got in touch with an expert, researched the
> > feasibility of such a goal and how much it would cost if I wanted it to
> be
> > done by somebody else.
> >
> > If you have other ideas on how we can address "*the tooling problem"*, I
> am
> > all ears.
> >
> > *P.S.* I am changing the subject of the thread, since I think it makes
> the
> > issue personal, which I believe wasn't your intention.
> >
> > On Wed, Jun 16, 2021 at 9:55 AM Gary Gregory <ga...@gmail.com>
> > wrote:
> >
> > > Volkan Yazıcı (@yazicivo) tweeted at 3:34 PM on Sun, Jun 13, 2021:
> > > Do you know of a Bazel ninja who can migrate Log4j's Maven builds? I
> can
> > > offer €2k (and maybe more) for such a service.
> > > (https://twitter.com/yazicivo/status/1404160220025217027?s=03)
> > >
> > > How do we maintain something like this? Sounds like a terrible idea to
> > me.
> > >
> > > Gary
> > >
> >
>
Re: Bazel for builds
Posted by Jeanderson <je...@gmail.com>.
Hi all,
I'm not particularly involved with Log4J development but I've been
following this list for a few months now just because I'm curious about
Log4J development (and also because I'm also doing some R&D related to
logging). In fact, this is my first ever post here.
May I ask some clarifying questions?
- What are the ongoing struggles that make Maven not a good fit for Log4J?
- What are you expecting to achieve by switching to Bazel?
I'm genuinely interested in this discussion and I hope that it may be
beneficial here.
Apologies in advance if this discussion is not convenient for this thread.
In case you are wondering, no, I have no connections with Maven or Bazel.
Best,
Jeanderson Candido
Em qua., 16 de jun. de 2021 às 10:28, Volkan Yazıcı <vo...@gmail.com>
escreveu:
> *[First and foremost, this is my personal Tweet and there I explicitly
> stated that "*I* can offer" the amount for such a work. After completing
> such a project, we need to have a consensus within us to merge it. I want
> to explicitly avoid any unintended misunderstandings.]*
>
> Gary, it was you who said "If the tooling is a problem, then let's look at
> that first." I definitely had thought the same and evaluated Bazel myself
> for quite some time a couple of months ago. Both Maven and Gradle have
> quite some limitations for monorepos similar to Log4j. I agree that Bazel
> is a total stranger to us (hence the maintenance problem), IDE support lags
> behind significantly, etc. Yet, it is a tool designed for fast, correct,
> and reproducible build of monorepos – a tool that is so-called to be a
> perfect fit for us. My personal evaluation of Bazel did not yield a
> positive outcome. Before crossing out this option, I wanted to ask around
> to some Bazel experts for their feedback and that is what I did. In
> summary, the outcome was again negative – which I was intending to discuss
> in detail in the upcoming online meeting.
>
> Long story short, rather than complaining, I gave it a spin myself. It
> wasn't convincing enough, I got in touch with an expert, researched the
> feasibility of such a goal and how much it would cost if I wanted it to be
> done by somebody else.
>
> If you have other ideas on how we can address "*the tooling problem"*, I am
> all ears.
>
> *P.S.* I am changing the subject of the thread, since I think it makes the
> issue personal, which I believe wasn't your intention.
>
> On Wed, Jun 16, 2021 at 9:55 AM Gary Gregory <ga...@gmail.com>
> wrote:
>
> > Volkan Yazıcı (@yazicivo) tweeted at 3:34 PM on Sun, Jun 13, 2021:
> > Do you know of a Bazel ninja who can migrate Log4j's Maven builds? I can
> > offer €2k (and maybe more) for such a service.
> > (https://twitter.com/yazicivo/status/1404160220025217027?s=03)
> >
> > How do we maintain something like this? Sounds like a terrible idea to
> me.
> >
> > Gary
> >
>