You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by Gary Gregory <ga...@gmail.com> on 2022/05/01 11:54:29 UTC
Re: Upgradable dependencies
Hi Volkan,
I see how you build and how you do version control as orthogonal concerns.
Currently we build and deliver everything from the root of the git checkout
directory. It does not have to be that way is all I'm saying. One could
just cd to a sub directory and work there. You could still have the option
to build the world from the root as a convenient way to make sure
everything works together and there is no breakage.
https://en.m.wikipedia.org/wiki/Monorepo
Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9] all
employ very large monorepos with varying strategies to scale build systems
and version control software with a large volume of code and daily changes.
Log4j is tiny in comparison.
Gary
On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <vo...@yazi.ci> wrote:
> I disagree that multi-repo setup is a recipe for disaster. It works
> perfectly well for Maven. Actually their setup is way more complicated:
> dedicated repo for each module! But let's assume you're right and
> multi-repos decrease the shelf time. Isn't that something good? I like
> repos that rot, it gives a clear message about that project: it simply is
> dead. If we migrate to multiple repos and some never get to make a release,
> what does this mean? Apparently nothing interesting is happening here. If
> there is a demand from the community to warrant a release, I am sure that
> release will come. After all, the very same community attempted to
> resurrect Log4j 1.
>
> I also like single repo setups where build and test times are within
> bearable limits, though Log4j is not one of them. Note that multi-repo
> discussion started, because we don't know any other way to solve the
> current set of problems. If you can think of how to address these concerns
> in some other way (e.g., "move folders around"), I am all ears.
>
> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <ga...@gmail.com>
> wrote:
>
> > Breaking our mono repo into a bunch of smaller repo pieces is a recipe
> for
> > software rot and orphanage IMO.
> >
> > For example, we have never released out of the tools repo, and I have
> folks
> > at work at use code from that repo for some internal facing work. Now, I
> > could take the time and do a release of course but... priorities. OTOH, I
> > could have gotten a release for "free" if that code was in the mono repo
> (I
> > am not advocating to put it back).
> >
> > This email might be better articulated in a call but I am trying to be
> > constructive: I think we can keep our mono repo and develop faster by
> > reorganizing what we have now. I like having everything in one place
> > myself. I am not sure how to actually move folders around to achieve
> > whatever new release cycle/split up we want, but I am sure it is do able.
> >
> > Gary
> >
> > On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <vo...@yazi.ci> wrote:
> >
> > > I have raised this issue in the past. I even came up with a proposal on
> > the
> > > repository structure
> > > <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3> to
> be
> > > followed. I am a big proponent of breaking up the logging-log4j2
> > repository
> > > into multiple ones. The current size of the code base is simply
> > > incomprehensible and slows down the development cycle tremendously due
> to
> > > excessive compilation and test times.
> > >
> > > *The proposal*
> > >
> > > Let me share the break down I have in mind:
> > >
> > > *logging-log4j2-api*
> > > log4j-1.2-api
> > > log4j-api
> > >
> > > *logging-log4j2*
> > > log4j-bom
> > > log4j-core
> > > log4j-core-its
> > > log4j-gctests
> > > log4j-iostreams
> > > log4j-perf
> > > log4j-plugins
> > >
> > > *logging-log4j2-appender*
> > > log4j-cassandra
> > > log4j-couchdb
> > > log4j-csv
> > > log4j-flume-ng
> > > log4j-mongodb3
> > > log4j-mongodb4
> > > log4j-smtp
> > >
> > > *logging-log4j2-appender-jdbc*
> > > log4j-jdbc (?)
> > > log4j-jdbc-dbcp2 (?)
> > > log4j-jpa
> > >
> > > *logging-log4j2-appender-queue*
> > > log4j-jeromq
> > > log4j-jms
> > > log4j-kafka
> > > log4j-redis
> > >
> > > *logging-log4j2-binding*
> > > log4j-jcl
> > > log4j-jpl
> > > log4j-jul
> > > log4j-liquibase
> > > log4j-slf4j18-impl
> > > log4j-slf4j-impl
> > > log4j-to-slf4j
> > >
> > > *logging-log4j2-container*
> > > log4j-docker
> > > log4j-kubernetes
> > >
> > > *logging-log4j2-gui*
> > > log4j-jmx-gui
> > >
> > > *logging-log4j2-jee*
> > > log4j-appserver
> > > log4j-taglib
> > > log4j-web
> > >
> > > *logging-log4j2-layout-json*
> > > log4j-layout-gelf
> > > log4j-layout-jackson
> > > log4j-layout-jackson-json
> > > log4j-layout-jackson-xml
> > > log4j-layout-jackson-yaml
> > > log4j-layout-template-json
> > >
> > > *logging-log4j2-osgi*
> > > log4j-osgi
> > >
> > > *logging-log4j2-spring*
> > > log4j-spring-boot
> > > log4j-spring-cloud-config
> > >
> > > Note that above breakdown contains every available module in master,
> > except
> > > log4j-samples, which has the following sub-modules:
> > >
> > > log4j-samples-loggerProperties (contains 2 very trivial example
> classes,
> > > log4j-core [tests] contains more comprehensive alternatives)
> > > log4j-samples-flume-remote (flume-specific)
> > > log4j-samples-flume-embedded (flume-specific)
> > > log4j-samples-flume-common (flume-specific)
> > > log4j-samples-configuration (contains 3 very trivial example classes,
> > > log4j-core [tests] contains more comprehensive alternatives)
> > >
> > > I propose removing the log4j-samples module and, if necessary, moving
> > > Flume-related parts to the log4j-flume-ng.
> > >
> > > *The problem with multiple repositories*
> > >
> > > I made this proposal in June 2021. Since then the more my familiarity
> > with
> > > the website and manual generation increased, the less I became inclined
> > to
> > > move forward. I am not much concerned about the increased release labor
> > the
> > > Ralph has mentioned – a majority of these repos will hardly get an
> update
> > > to warrant a release. Though the fact that current website generation
> > > depends on the presence of these modules to be in the same repository
> and
> > > with multiple repositories, the manual and source codes will be in
> > > different repositories... These concern me a lot. I think, the steps to
> > > overhaul the project structure should better be in the following order:
> > >
> > > 1. Replace `site` phase (`site` phase won't be needed anymore.
> Website
> > > will independently be generated in a single step via JBake.)
> > > 2. Rewrite the manual
> > > 3. Migrate to a multi-repo structure (Each repo will have its own
> > > website and manual, ala Maven. Here the code in steps 1 and 2 will
> be
> > > reused.)
> > >
> > >
> > >
> > > On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers <
> ralph.goers@dslextreme.com>
> > > wrote:
> > >
> > > > I wouldn’t necessarily want to penalize them. At the same time, I
> don’t
> > > > think we want a ton of extra repos each of which needs its own
> release.
> > > >
> > > > Ralph
> > > >
> > > > > On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz <
> > > piotr.karwasz@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <vo...@yazi.ci>
> wrote:
> > > > >
> > > > >> +1 with moving Cassandra et al. to their own repo
> > > > >>
> > > > >
> > > > > I agree: Cassandra's tests were one of the most common reasons for
> > > > failing
> > > > > CI.
> > > > > Maybe it could share a repository with other database appenders.
> > > > >
> > > > > Piotr
> > > >
> > > >
> > >
> >
>
Re: Upgradable dependencies
Posted by Ralph Goers <ra...@dslextreme.com>.
I really don’t see how this helps. We aren’t going to release subcomponents of a Git repo.
I wouldn’t even begin to know how to do that at the ASF.
The problems I have are all related to how long it takes to prepare and perform a release.
I can definitely say that I have encountered many times where builds haven’t worked because
people are only building the things they changed without realizing they were impacting
other components, and they do this because it is simply too time consuming to run a full
build. While separating the components won’t fix this, it will make it so that what is being
released has been tested. So, for example, if a change in core breaks something in an
Appender in another module that will be fixed in that component before its next release.
The main reason we don’t do frequent releases is because I need to find 2 or 3 free
days to be able to do it. This is largely because I go through the issues and look for stuff
we broke in previous releases - we have 3 or 4 of those right now, and I build the web site
usually 4 or 5 times because the site build always has problems and no one ever runs it
except when I do it during a release.
So for me, the two key items are:
1. Stop using Maven site and build the web site in logging-log4j2-site using a much
simplified process.
2. Break up the build into smaller pieces so that full development builds can be done
on committers laptops in no more than 5 minutes and that the release build doesn’t take
much longer than that.
Although we discuss parallel builds, to date every attempt to do that has failed. The
effort to go through all the tests and verify that they have no side effects that could
impact other tests seems to be more than any of us can do. If we can’t even get
log4j-api to build without problems I don’t see how there is any hope for log4j-core
to do that.
Ralph
> On May 1, 2022, at 1:54 PM, Gary Gregory <ga...@gmail.com> wrote:
>
> Hi Volkan,
>
> I see how you build and how you do version control as orthogonal concerns.
> Currently we build and deliver everything from the root of the git checkout
> directory. It does not have to be that way is all I'm saying. One could
> just cd to a sub directory and work there. You could still have the option
> to build the world from the root as a convenient way to make sure
> everything works together and there is no breakage.
>
> https://en.m.wikipedia.org/wiki/Monorepo
>
> Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9] all
> employ very large monorepos with varying strategies to scale build systems
> and version control software with a large volume of code and daily changes.
>
> Log4j is tiny in comparison.
>
> Gary
>
> On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <vo...@yazi.ci> wrote:
>
>> I disagree that multi-repo setup is a recipe for disaster. It works
>> perfectly well for Maven. Actually their setup is way more complicated:
>> dedicated repo for each module! But let's assume you're right and
>> multi-repos decrease the shelf time. Isn't that something good? I like
>> repos that rot, it gives a clear message about that project: it simply is
>> dead. If we migrate to multiple repos and some never get to make a release,
>> what does this mean? Apparently nothing interesting is happening here. If
>> there is a demand from the community to warrant a release, I am sure that
>> release will come. After all, the very same community attempted to
>> resurrect Log4j 1.
>>
>> I also like single repo setups where build and test times are within
>> bearable limits, though Log4j is not one of them. Note that multi-repo
>> discussion started, because we don't know any other way to solve the
>> current set of problems. If you can think of how to address these concerns
>> in some other way (e.g., "move folders around"), I am all ears.
>>
>> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <ga...@gmail.com>
>> wrote:
>>
>>> Breaking our mono repo into a bunch of smaller repo pieces is a recipe
>> for
>>> software rot and orphanage IMO.
>>>
>>> For example, we have never released out of the tools repo, and I have
>> folks
>>> at work at use code from that repo for some internal facing work. Now, I
>>> could take the time and do a release of course but... priorities. OTOH, I
>>> could have gotten a release for "free" if that code was in the mono repo
>> (I
>>> am not advocating to put it back).
>>>
>>> This email might be better articulated in a call but I am trying to be
>>> constructive: I think we can keep our mono repo and develop faster by
>>> reorganizing what we have now. I like having everything in one place
>>> myself. I am not sure how to actually move folders around to achieve
>>> whatever new release cycle/split up we want, but I am sure it is do able.
>>>
>>> Gary
>>>
>>> On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <vo...@yazi.ci> wrote:
>>>
>>>> I have raised this issue in the past. I even came up with a proposal on
>>> the
>>>> repository structure
>>>> <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3> to
>> be
>>>> followed. I am a big proponent of breaking up the logging-log4j2
>>> repository
>>>> into multiple ones. The current size of the code base is simply
>>>> incomprehensible and slows down the development cycle tremendously due
>> to
>>>> excessive compilation and test times.
>>>>
>>>> *The proposal*
>>>>
>>>> Let me share the break down I have in mind:
>>>>
>>>> *logging-log4j2-api*
>>>> log4j-1.2-api
>>>> log4j-api
>>>>
>>>> *logging-log4j2*
>>>> log4j-bom
>>>> log4j-core
>>>> log4j-core-its
>>>> log4j-gctests
>>>> log4j-iostreams
>>>> log4j-perf
>>>> log4j-plugins
>>>>
>>>> *logging-log4j2-appender*
>>>> log4j-cassandra
>>>> log4j-couchdb
>>>> log4j-csv
>>>> log4j-flume-ng
>>>> log4j-mongodb3
>>>> log4j-mongodb4
>>>> log4j-smtp
>>>>
>>>> *logging-log4j2-appender-jdbc*
>>>> log4j-jdbc (?)
>>>> log4j-jdbc-dbcp2 (?)
>>>> log4j-jpa
>>>>
>>>> *logging-log4j2-appender-queue*
>>>> log4j-jeromq
>>>> log4j-jms
>>>> log4j-kafka
>>>> log4j-redis
>>>>
>>>> *logging-log4j2-binding*
>>>> log4j-jcl
>>>> log4j-jpl
>>>> log4j-jul
>>>> log4j-liquibase
>>>> log4j-slf4j18-impl
>>>> log4j-slf4j-impl
>>>> log4j-to-slf4j
>>>>
>>>> *logging-log4j2-container*
>>>> log4j-docker
>>>> log4j-kubernetes
>>>>
>>>> *logging-log4j2-gui*
>>>> log4j-jmx-gui
>>>>
>>>> *logging-log4j2-jee*
>>>> log4j-appserver
>>>> log4j-taglib
>>>> log4j-web
>>>>
>>>> *logging-log4j2-layout-json*
>>>> log4j-layout-gelf
>>>> log4j-layout-jackson
>>>> log4j-layout-jackson-json
>>>> log4j-layout-jackson-xml
>>>> log4j-layout-jackson-yaml
>>>> log4j-layout-template-json
>>>>
>>>> *logging-log4j2-osgi*
>>>> log4j-osgi
>>>>
>>>> *logging-log4j2-spring*
>>>> log4j-spring-boot
>>>> log4j-spring-cloud-config
>>>>
>>>> Note that above breakdown contains every available module in master,
>>> except
>>>> log4j-samples, which has the following sub-modules:
>>>>
>>>> log4j-samples-loggerProperties (contains 2 very trivial example
>> classes,
>>>> log4j-core [tests] contains more comprehensive alternatives)
>>>> log4j-samples-flume-remote (flume-specific)
>>>> log4j-samples-flume-embedded (flume-specific)
>>>> log4j-samples-flume-common (flume-specific)
>>>> log4j-samples-configuration (contains 3 very trivial example classes,
>>>> log4j-core [tests] contains more comprehensive alternatives)
>>>>
>>>> I propose removing the log4j-samples module and, if necessary, moving
>>>> Flume-related parts to the log4j-flume-ng.
>>>>
>>>> *The problem with multiple repositories*
>>>>
>>>> I made this proposal in June 2021. Since then the more my familiarity
>>> with
>>>> the website and manual generation increased, the less I became inclined
>>> to
>>>> move forward. I am not much concerned about the increased release labor
>>> the
>>>> Ralph has mentioned – a majority of these repos will hardly get an
>> update
>>>> to warrant a release. Though the fact that current website generation
>>>> depends on the presence of these modules to be in the same repository
>> and
>>>> with multiple repositories, the manual and source codes will be in
>>>> different repositories... These concern me a lot. I think, the steps to
>>>> overhaul the project structure should better be in the following order:
>>>>
>>>> 1. Replace `site` phase (`site` phase won't be needed anymore.
>> Website
>>>> will independently be generated in a single step via JBake.)
>>>> 2. Rewrite the manual
>>>> 3. Migrate to a multi-repo structure (Each repo will have its own
>>>> website and manual, ala Maven. Here the code in steps 1 and 2 will
>> be
>>>> reused.)
>>>>
>>>>
>>>>
>>>> On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers <
>> ralph.goers@dslextreme.com>
>>>> wrote:
>>>>
>>>>> I wouldn’t necessarily want to penalize them. At the same time, I
>> don’t
>>>>> think we want a ton of extra repos each of which needs its own
>> release.
>>>>>
>>>>> Ralph
>>>>>
>>>>>> On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz <
>>>> piotr.karwasz@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <vo...@yazi.ci>
>> wrote:
>>>>>>
>>>>>>> +1 with moving Cassandra et al. to their own repo
>>>>>>>
>>>>>>
>>>>>> I agree: Cassandra's tests were one of the most common reasons for
>>>>> failing
>>>>>> CI.
>>>>>> Maybe it could share a repository with other database appenders.
>>>>>>
>>>>>> Piotr
>>>>>
>>>>>
>>>>
>>>
>>
Re: Upgradable dependencies
Posted by Gary Gregory <ga...@gmail.com>.
I wonder the Java compiler take advantage for multiple cores/threads?
Gary
On Sun, May 1, 2022, 13:11 Matt Sicker <bo...@gmail.com> wrote:
> So I think we’re back at the build optimization problem. If we could
> enable more parallelism there, that could help a lot as most of the modules
> compile and test within a few seconds each.
>
> —
> Matt Sicker
>
> > On May 1, 2022, at 06:56, Gary Gregory <ga...@gmail.com> wrote:
> >
> > And yes, one repo per module is another option, a la Maven as you point
> > out, but what is being proposed is some halfway solution which to me
> feels
> > confusing and arbitrary just because today this and that module is in
> favor
> > but that some others.
> >
> > Gaty
> >
> >> On Sun, May 1, 2022, 07:54 Gary Gregory <ga...@gmail.com> wrote:
> >>
> >> Hi Volkan,
> >>
> >> I see how you build and how you do version control as orthogonal
> concerns.
> >> Currently we build and deliver everything from the root of the git
> checkout
> >> directory. It does not have to be that way is all I'm saying. One could
> >> just cd to a sub directory and work there. You could still have the
> option
> >> to build the world from the root as a convenient way to make sure
> >> everything works together and there is no breakage.
> >>
> >> https://en.m.wikipedia.org/wiki/Monorepo
> >>
> >> Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9]
> all
> >> employ very large monorepos with varying strategies to scale build
> systems
> >> and version control software with a large volume of code and daily
> changes.
> >>
> >> Log4j is tiny in comparison.
> >>
> >> Gary
> >>
> >>> On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <vo...@yazi.ci> wrote:
> >>>
> >>> I disagree that multi-repo setup is a recipe for disaster. It works
> >>> perfectly well for Maven. Actually their setup is way more complicated:
> >>> dedicated repo for each module! But let's assume you're right and
> >>> multi-repos decrease the shelf time. Isn't that something good? I like
> >>> repos that rot, it gives a clear message about that project: it simply
> is
> >>> dead. If we migrate to multiple repos and some never get to make a
> >>> release,
> >>> what does this mean? Apparently nothing interesting is happening here.
> If
> >>> there is a demand from the community to warrant a release, I am sure
> that
> >>> release will come. After all, the very same community attempted to
> >>> resurrect Log4j 1.
> >>>
> >>> I also like single repo setups where build and test times are within
> >>> bearable limits, though Log4j is not one of them. Note that multi-repo
> >>> discussion started, because we don't know any other way to solve the
> >>> current set of problems. If you can think of how to address these
> concerns
> >>> in some other way (e.g., "move folders around"), I am all ears.
> >>>
> >>> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <ga...@gmail.com>
> >>> wrote:
> >>>
> >>>> Breaking our mono repo into a bunch of smaller repo pieces is a recipe
> >>> for
> >>>> software rot and orphanage IMO.
> >>>>
> >>>> For example, we have never released out of the tools repo, and I have
> >>> folks
> >>>> at work at use code from that repo for some internal facing work.
> Now, I
> >>>> could take the time and do a release of course but... priorities.
> OTOH,
> >>> I
> >>>> could have gotten a release for "free" if that code was in the mono
> >>> repo (I
> >>>> am not advocating to put it back).
> >>>>
> >>>> This email might be better articulated in a call but I am trying to be
> >>>> constructive: I think we can keep our mono repo and develop faster by
> >>>> reorganizing what we have now. I like having everything in one place
> >>>> myself. I am not sure how to actually move folders around to achieve
> >>>> whatever new release cycle/split up we want, but I am sure it is do
> >>> able.
> >>>>
> >>>> Gary
> >>>>
> >>>> On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <vo...@yazi.ci> wrote:
> >>>>
> >>>>> I have raised this issue in the past. I even came up with a proposal
> >>> on
> >>>> the
> >>>>> repository structure
> >>>>> <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3>
> >>> to be
> >>>>> followed. I am a big proponent of breaking up the logging-log4j2
> >>>> repository
> >>>>> into multiple ones. The current size of the code base is simply
> >>>>> incomprehensible and slows down the development cycle tremendously
> >>> due to
> >>>>> excessive compilation and test times.
> >>>>>
> >>>>> *The proposal*
> >>>>>
> >>>>> Let me share the break down I have in mind:
> >>>>>
> >>>>> *logging-log4j2-api*
> >>>>> log4j-1.2-api
> >>>>> log4j-api
> >>>>>
> >>>>> *logging-log4j2*
> >>>>> log4j-bom
> >>>>> log4j-core
> >>>>> log4j-core-its
> >>>>> log4j-gctests
> >>>>> log4j-iostreams
> >>>>> log4j-perf
> >>>>> log4j-plugins
> >>>>>
> >>>>> *logging-log4j2-appender*
> >>>>> log4j-cassandra
> >>>>> log4j-couchdb
> >>>>> log4j-csv
> >>>>> log4j-flume-ng
> >>>>> log4j-mongodb3
> >>>>> log4j-mongodb4
> >>>>> log4j-smtp
> >>>>>
> >>>>> *logging-log4j2-appender-jdbc*
> >>>>> log4j-jdbc (?)
> >>>>> log4j-jdbc-dbcp2 (?)
> >>>>> log4j-jpa
> >>>>>
> >>>>> *logging-log4j2-appender-queue*
> >>>>> log4j-jeromq
> >>>>> log4j-jms
> >>>>> log4j-kafka
> >>>>> log4j-redis
> >>>>>
> >>>>> *logging-log4j2-binding*
> >>>>> log4j-jcl
> >>>>> log4j-jpl
> >>>>> log4j-jul
> >>>>> log4j-liquibase
> >>>>> log4j-slf4j18-impl
> >>>>> log4j-slf4j-impl
> >>>>> log4j-to-slf4j
> >>>>>
> >>>>> *logging-log4j2-container*
> >>>>> log4j-docker
> >>>>> log4j-kubernetes
> >>>>>
> >>>>> *logging-log4j2-gui*
> >>>>> log4j-jmx-gui
> >>>>>
> >>>>> *logging-log4j2-jee*
> >>>>> log4j-appserver
> >>>>> log4j-taglib
> >>>>> log4j-web
> >>>>>
> >>>>> *logging-log4j2-layout-json*
> >>>>> log4j-layout-gelf
> >>>>> log4j-layout-jackson
> >>>>> log4j-layout-jackson-json
> >>>>> log4j-layout-jackson-xml
> >>>>> log4j-layout-jackson-yaml
> >>>>> log4j-layout-template-json
> >>>>>
> >>>>> *logging-log4j2-osgi*
> >>>>> log4j-osgi
> >>>>>
> >>>>> *logging-log4j2-spring*
> >>>>> log4j-spring-boot
> >>>>> log4j-spring-cloud-config
> >>>>>
> >>>>> Note that above breakdown contains every available module in master,
> >>>> except
> >>>>> log4j-samples, which has the following sub-modules:
> >>>>>
> >>>>> log4j-samples-loggerProperties (contains 2 very trivial example
> >>> classes,
> >>>>> log4j-core [tests] contains more comprehensive alternatives)
> >>>>> log4j-samples-flume-remote (flume-specific)
> >>>>> log4j-samples-flume-embedded (flume-specific)
> >>>>> log4j-samples-flume-common (flume-specific)
> >>>>> log4j-samples-configuration (contains 3 very trivial example classes,
> >>>>> log4j-core [tests] contains more comprehensive alternatives)
> >>>>>
> >>>>> I propose removing the log4j-samples module and, if necessary, moving
> >>>>> Flume-related parts to the log4j-flume-ng.
> >>>>>
> >>>>> *The problem with multiple repositories*
> >>>>>
> >>>>> I made this proposal in June 2021. Since then the more my familiarity
> >>>> with
> >>>>> the website and manual generation increased, the less I became
> >>> inclined
> >>>> to
> >>>>> move forward. I am not much concerned about the increased release
> >>> labor
> >>>> the
> >>>>> Ralph has mentioned – a majority of these repos will hardly get an
> >>> update
> >>>>> to warrant a release. Though the fact that current website generation
> >>>>> depends on the presence of these modules to be in the same repository
> >>> and
> >>>>> with multiple repositories, the manual and source codes will be in
> >>>>> different repositories... These concern me a lot. I think, the steps
> >>> to
> >>>>> overhaul the project structure should better be in the following
> >>> order:
> >>>>>
> >>>>> 1. Replace `site` phase (`site` phase won't be needed anymore.
> >>> Website
> >>>>> will independently be generated in a single step via JBake.)
> >>>>> 2. Rewrite the manual
> >>>>> 3. Migrate to a multi-repo structure (Each repo will have its own
> >>>>> website and manual, ala Maven. Here the code in steps 1 and 2 will
> >>> be
> >>>>> reused.)
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers <
> >>> ralph.goers@dslextreme.com>
> >>>>> wrote:
> >>>>>
> >>>>>> I wouldn’t necessarily want to penalize them. At the same time, I
> >>> don’t
> >>>>>> think we want a ton of extra repos each of which needs its own
> >>> release.
> >>>>>>
> >>>>>> Ralph
> >>>>>>
> >>>>>>> On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz <
> >>>>> piotr.karwasz@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <vo...@yazi.ci>
> >>> wrote:
> >>>>>>>
> >>>>>>>> +1 with moving Cassandra et al. to their own repo
> >>>>>>>>
> >>>>>>>
> >>>>>>> I agree: Cassandra's tests were one of the most common reasons for
> >>>>>> failing
> >>>>>>> CI.
> >>>>>>> Maybe it could share a repository with other database appenders.
> >>>>>>>
> >>>>>>> Piotr
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
Re: Upgradable dependencies
Posted by Matt Sicker <bo...@gmail.com>.
So I think we’re back at the build optimization problem. If we could enable more parallelism there, that could help a lot as most of the modules compile and test within a few seconds each.
—
Matt Sicker
> On May 1, 2022, at 06:56, Gary Gregory <ga...@gmail.com> wrote:
>
> And yes, one repo per module is another option, a la Maven as you point
> out, but what is being proposed is some halfway solution which to me feels
> confusing and arbitrary just because today this and that module is in favor
> but that some others.
>
> Gaty
>
>> On Sun, May 1, 2022, 07:54 Gary Gregory <ga...@gmail.com> wrote:
>>
>> Hi Volkan,
>>
>> I see how you build and how you do version control as orthogonal concerns.
>> Currently we build and deliver everything from the root of the git checkout
>> directory. It does not have to be that way is all I'm saying. One could
>> just cd to a sub directory and work there. You could still have the option
>> to build the world from the root as a convenient way to make sure
>> everything works together and there is no breakage.
>>
>> https://en.m.wikipedia.org/wiki/Monorepo
>>
>> Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9] all
>> employ very large monorepos with varying strategies to scale build systems
>> and version control software with a large volume of code and daily changes.
>>
>> Log4j is tiny in comparison.
>>
>> Gary
>>
>>> On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <vo...@yazi.ci> wrote:
>>>
>>> I disagree that multi-repo setup is a recipe for disaster. It works
>>> perfectly well for Maven. Actually their setup is way more complicated:
>>> dedicated repo for each module! But let's assume you're right and
>>> multi-repos decrease the shelf time. Isn't that something good? I like
>>> repos that rot, it gives a clear message about that project: it simply is
>>> dead. If we migrate to multiple repos and some never get to make a
>>> release,
>>> what does this mean? Apparently nothing interesting is happening here. If
>>> there is a demand from the community to warrant a release, I am sure that
>>> release will come. After all, the very same community attempted to
>>> resurrect Log4j 1.
>>>
>>> I also like single repo setups where build and test times are within
>>> bearable limits, though Log4j is not one of them. Note that multi-repo
>>> discussion started, because we don't know any other way to solve the
>>> current set of problems. If you can think of how to address these concerns
>>> in some other way (e.g., "move folders around"), I am all ears.
>>>
>>> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <ga...@gmail.com>
>>> wrote:
>>>
>>>> Breaking our mono repo into a bunch of smaller repo pieces is a recipe
>>> for
>>>> software rot and orphanage IMO.
>>>>
>>>> For example, we have never released out of the tools repo, and I have
>>> folks
>>>> at work at use code from that repo for some internal facing work. Now, I
>>>> could take the time and do a release of course but... priorities. OTOH,
>>> I
>>>> could have gotten a release for "free" if that code was in the mono
>>> repo (I
>>>> am not advocating to put it back).
>>>>
>>>> This email might be better articulated in a call but I am trying to be
>>>> constructive: I think we can keep our mono repo and develop faster by
>>>> reorganizing what we have now. I like having everything in one place
>>>> myself. I am not sure how to actually move folders around to achieve
>>>> whatever new release cycle/split up we want, but I am sure it is do
>>> able.
>>>>
>>>> Gary
>>>>
>>>> On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <vo...@yazi.ci> wrote:
>>>>
>>>>> I have raised this issue in the past. I even came up with a proposal
>>> on
>>>> the
>>>>> repository structure
>>>>> <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3>
>>> to be
>>>>> followed. I am a big proponent of breaking up the logging-log4j2
>>>> repository
>>>>> into multiple ones. The current size of the code base is simply
>>>>> incomprehensible and slows down the development cycle tremendously
>>> due to
>>>>> excessive compilation and test times.
>>>>>
>>>>> *The proposal*
>>>>>
>>>>> Let me share the break down I have in mind:
>>>>>
>>>>> *logging-log4j2-api*
>>>>> log4j-1.2-api
>>>>> log4j-api
>>>>>
>>>>> *logging-log4j2*
>>>>> log4j-bom
>>>>> log4j-core
>>>>> log4j-core-its
>>>>> log4j-gctests
>>>>> log4j-iostreams
>>>>> log4j-perf
>>>>> log4j-plugins
>>>>>
>>>>> *logging-log4j2-appender*
>>>>> log4j-cassandra
>>>>> log4j-couchdb
>>>>> log4j-csv
>>>>> log4j-flume-ng
>>>>> log4j-mongodb3
>>>>> log4j-mongodb4
>>>>> log4j-smtp
>>>>>
>>>>> *logging-log4j2-appender-jdbc*
>>>>> log4j-jdbc (?)
>>>>> log4j-jdbc-dbcp2 (?)
>>>>> log4j-jpa
>>>>>
>>>>> *logging-log4j2-appender-queue*
>>>>> log4j-jeromq
>>>>> log4j-jms
>>>>> log4j-kafka
>>>>> log4j-redis
>>>>>
>>>>> *logging-log4j2-binding*
>>>>> log4j-jcl
>>>>> log4j-jpl
>>>>> log4j-jul
>>>>> log4j-liquibase
>>>>> log4j-slf4j18-impl
>>>>> log4j-slf4j-impl
>>>>> log4j-to-slf4j
>>>>>
>>>>> *logging-log4j2-container*
>>>>> log4j-docker
>>>>> log4j-kubernetes
>>>>>
>>>>> *logging-log4j2-gui*
>>>>> log4j-jmx-gui
>>>>>
>>>>> *logging-log4j2-jee*
>>>>> log4j-appserver
>>>>> log4j-taglib
>>>>> log4j-web
>>>>>
>>>>> *logging-log4j2-layout-json*
>>>>> log4j-layout-gelf
>>>>> log4j-layout-jackson
>>>>> log4j-layout-jackson-json
>>>>> log4j-layout-jackson-xml
>>>>> log4j-layout-jackson-yaml
>>>>> log4j-layout-template-json
>>>>>
>>>>> *logging-log4j2-osgi*
>>>>> log4j-osgi
>>>>>
>>>>> *logging-log4j2-spring*
>>>>> log4j-spring-boot
>>>>> log4j-spring-cloud-config
>>>>>
>>>>> Note that above breakdown contains every available module in master,
>>>> except
>>>>> log4j-samples, which has the following sub-modules:
>>>>>
>>>>> log4j-samples-loggerProperties (contains 2 very trivial example
>>> classes,
>>>>> log4j-core [tests] contains more comprehensive alternatives)
>>>>> log4j-samples-flume-remote (flume-specific)
>>>>> log4j-samples-flume-embedded (flume-specific)
>>>>> log4j-samples-flume-common (flume-specific)
>>>>> log4j-samples-configuration (contains 3 very trivial example classes,
>>>>> log4j-core [tests] contains more comprehensive alternatives)
>>>>>
>>>>> I propose removing the log4j-samples module and, if necessary, moving
>>>>> Flume-related parts to the log4j-flume-ng.
>>>>>
>>>>> *The problem with multiple repositories*
>>>>>
>>>>> I made this proposal in June 2021. Since then the more my familiarity
>>>> with
>>>>> the website and manual generation increased, the less I became
>>> inclined
>>>> to
>>>>> move forward. I am not much concerned about the increased release
>>> labor
>>>> the
>>>>> Ralph has mentioned – a majority of these repos will hardly get an
>>> update
>>>>> to warrant a release. Though the fact that current website generation
>>>>> depends on the presence of these modules to be in the same repository
>>> and
>>>>> with multiple repositories, the manual and source codes will be in
>>>>> different repositories... These concern me a lot. I think, the steps
>>> to
>>>>> overhaul the project structure should better be in the following
>>> order:
>>>>>
>>>>> 1. Replace `site` phase (`site` phase won't be needed anymore.
>>> Website
>>>>> will independently be generated in a single step via JBake.)
>>>>> 2. Rewrite the manual
>>>>> 3. Migrate to a multi-repo structure (Each repo will have its own
>>>>> website and manual, ala Maven. Here the code in steps 1 and 2 will
>>> be
>>>>> reused.)
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers <
>>> ralph.goers@dslextreme.com>
>>>>> wrote:
>>>>>
>>>>>> I wouldn’t necessarily want to penalize them. At the same time, I
>>> don’t
>>>>>> think we want a ton of extra repos each of which needs its own
>>> release.
>>>>>>
>>>>>> Ralph
>>>>>>
>>>>>>> On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz <
>>>>> piotr.karwasz@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <vo...@yazi.ci>
>>> wrote:
>>>>>>>
>>>>>>>> +1 with moving Cassandra et al. to their own repo
>>>>>>>>
>>>>>>>
>>>>>>> I agree: Cassandra's tests were one of the most common reasons for
>>>>>> failing
>>>>>>> CI.
>>>>>>> Maybe it could share a repository with other database appenders.
>>>>>>>
>>>>>>> Piotr
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: Upgradable dependencies
Posted by Matt Sicker <bo...@gmail.com>.
I tried to enable tests to parallelize themselves. While that’s a far more challenging task, I don’t think it should be that hard to figure out how to make the -T option in Maven work properly. Each module could be compiled in parallel while running tests sequentially within each module; that should work without code changes in theory. While that would still leave Core as a long running module, that should also reduce a bit as we start splitting that up further.
—
Matt Sicker
> On May 1, 2022, at 13:48, Ralph Goers <ra...@dslextreme.com> wrote:
>
> I believe there is a set of repos that does make sense. Volkan was on the right
> track but got a few things incorrect, which I previously called out.
>
> Ralph
>
>> On May 1, 2022, at 1:56 PM, Gary Gregory <ga...@gmail.com> wrote:
>>
>> And yes, one repo per module is another option, a la Maven as you point
>> out, but what is being proposed is some halfway solution which to me feels
>> confusing and arbitrary just because today this and that module is in favor
>> but that some others.
>>
>> Gaty
>>
>>> On Sun, May 1, 2022, 07:54 Gary Gregory <ga...@gmail.com> wrote:
>>>
>>> Hi Volkan,
>>>
>>> I see how you build and how you do version control as orthogonal concerns.
>>> Currently we build and deliver everything from the root of the git checkout
>>> directory. It does not have to be that way is all I'm saying. One could
>>> just cd to a sub directory and work there. You could still have the option
>>> to build the world from the root as a convenient way to make sure
>>> everything works together and there is no breakage.
>>>
>>> https://en.m.wikipedia.org/wiki/Monorepo
>>>
>>> Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9] all
>>> employ very large monorepos with varying strategies to scale build systems
>>> and version control software with a large volume of code and daily changes.
>>>
>>> Log4j is tiny in comparison.
>>>
>>> Gary
>>>
>>>> On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <vo...@yazi.ci> wrote:
>>>
>>>> I disagree that multi-repo setup is a recipe for disaster. It works
>>>> perfectly well for Maven. Actually their setup is way more complicated:
>>>> dedicated repo for each module! But let's assume you're right and
>>>> multi-repos decrease the shelf time. Isn't that something good? I like
>>>> repos that rot, it gives a clear message about that project: it simply is
>>>> dead. If we migrate to multiple repos and some never get to make a
>>>> release,
>>>> what does this mean? Apparently nothing interesting is happening here. If
>>>> there is a demand from the community to warrant a release, I am sure that
>>>> release will come. After all, the very same community attempted to
>>>> resurrect Log4j 1.
>>>>
>>>> I also like single repo setups where build and test times are within
>>>> bearable limits, though Log4j is not one of them. Note that multi-repo
>>>> discussion started, because we don't know any other way to solve the
>>>> current set of problems. If you can think of how to address these concerns
>>>> in some other way (e.g., "move folders around"), I am all ears.
>>>>
>>>> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>> Breaking our mono repo into a bunch of smaller repo pieces is a recipe
>>>> for
>>>>> software rot and orphanage IMO.
>>>>>
>>>>> For example, we have never released out of the tools repo, and I have
>>>> folks
>>>>> at work at use code from that repo for some internal facing work. Now, I
>>>>> could take the time and do a release of course but... priorities. OTOH,
>>>> I
>>>>> could have gotten a release for "free" if that code was in the mono
>>>> repo (I
>>>>> am not advocating to put it back).
>>>>>
>>>>> This email might be better articulated in a call but I am trying to be
>>>>> constructive: I think we can keep our mono repo and develop faster by
>>>>> reorganizing what we have now. I like having everything in one place
>>>>> myself. I am not sure how to actually move folders around to achieve
>>>>> whatever new release cycle/split up we want, but I am sure it is do
>>>> able.
>>>>>
>>>>> Gary
>>>>>
>>>>> On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <vo...@yazi.ci> wrote:
>>>>>
>>>>>> I have raised this issue in the past. I even came up with a proposal
>>>> on
>>>>> the
>>>>>> repository structure
>>>>>> <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3>
>>>> to be
>>>>>> followed. I am a big proponent of breaking up the logging-log4j2
>>>>> repository
>>>>>> into multiple ones. The current size of the code base is simply
>>>>>> incomprehensible and slows down the development cycle tremendously
>>>> due to
>>>>>> excessive compilation and test times.
>>>>>>
>>>>>> *The proposal*
>>>>>>
>>>>>> Let me share the break down I have in mind:
>>>>>>
>>>>>> *logging-log4j2-api*
>>>>>> log4j-1.2-api
>>>>>> log4j-api
>>>>>>
>>>>>> *logging-log4j2*
>>>>>> log4j-bom
>>>>>> log4j-core
>>>>>> log4j-core-its
>>>>>> log4j-gctests
>>>>>> log4j-iostreams
>>>>>> log4j-perf
>>>>>> log4j-plugins
>>>>>>
>>>>>> *logging-log4j2-appender*
>>>>>> log4j-cassandra
>>>>>> log4j-couchdb
>>>>>> log4j-csv
>>>>>> log4j-flume-ng
>>>>>> log4j-mongodb3
>>>>>> log4j-mongodb4
>>>>>> log4j-smtp
>>>>>>
>>>>>> *logging-log4j2-appender-jdbc*
>>>>>> log4j-jdbc (?)
>>>>>> log4j-jdbc-dbcp2 (?)
>>>>>> log4j-jpa
>>>>>>
>>>>>> *logging-log4j2-appender-queue*
>>>>>> log4j-jeromq
>>>>>> log4j-jms
>>>>>> log4j-kafka
>>>>>> log4j-redis
>>>>>>
>>>>>> *logging-log4j2-binding*
>>>>>> log4j-jcl
>>>>>> log4j-jpl
>>>>>> log4j-jul
>>>>>> log4j-liquibase
>>>>>> log4j-slf4j18-impl
>>>>>> log4j-slf4j-impl
>>>>>> log4j-to-slf4j
>>>>>>
>>>>>> *logging-log4j2-container*
>>>>>> log4j-docker
>>>>>> log4j-kubernetes
>>>>>>
>>>>>> *logging-log4j2-gui*
>>>>>> log4j-jmx-gui
>>>>>>
>>>>>> *logging-log4j2-jee*
>>>>>> log4j-appserver
>>>>>> log4j-taglib
>>>>>> log4j-web
>>>>>>
>>>>>> *logging-log4j2-layout-json*
>>>>>> log4j-layout-gelf
>>>>>> log4j-layout-jackson
>>>>>> log4j-layout-jackson-json
>>>>>> log4j-layout-jackson-xml
>>>>>> log4j-layout-jackson-yaml
>>>>>> log4j-layout-template-json
>>>>>>
>>>>>> *logging-log4j2-osgi*
>>>>>> log4j-osgi
>>>>>>
>>>>>> *logging-log4j2-spring*
>>>>>> log4j-spring-boot
>>>>>> log4j-spring-cloud-config
>>>>>>
>>>>>> Note that above breakdown contains every available module in master,
>>>>> except
>>>>>> log4j-samples, which has the following sub-modules:
>>>>>>
>>>>>> log4j-samples-loggerProperties (contains 2 very trivial example
>>>> classes,
>>>>>> log4j-core [tests] contains more comprehensive alternatives)
>>>>>> log4j-samples-flume-remote (flume-specific)
>>>>>> log4j-samples-flume-embedded (flume-specific)
>>>>>> log4j-samples-flume-common (flume-specific)
>>>>>> log4j-samples-configuration (contains 3 very trivial example classes,
>>>>>> log4j-core [tests] contains more comprehensive alternatives)
>>>>>>
>>>>>> I propose removing the log4j-samples module and, if necessary, moving
>>>>>> Flume-related parts to the log4j-flume-ng.
>>>>>>
>>>>>> *The problem with multiple repositories*
>>>>>>
>>>>>> I made this proposal in June 2021. Since then the more my familiarity
>>>>> with
>>>>>> the website and manual generation increased, the less I became
>>>> inclined
>>>>> to
>>>>>> move forward. I am not much concerned about the increased release
>>>> labor
>>>>> the
>>>>>> Ralph has mentioned – a majority of these repos will hardly get an
>>>> update
>>>>>> to warrant a release. Though the fact that current website generation
>>>>>> depends on the presence of these modules to be in the same repository
>>>> and
>>>>>> with multiple repositories, the manual and source codes will be in
>>>>>> different repositories... These concern me a lot. I think, the steps
>>>> to
>>>>>> overhaul the project structure should better be in the following
>>>> order:
>>>>>>
>>>>>> 1. Replace `site` phase (`site` phase won't be needed anymore.
>>>> Website
>>>>>> will independently be generated in a single step via JBake.)
>>>>>> 2. Rewrite the manual
>>>>>> 3. Migrate to a multi-repo structure (Each repo will have its own
>>>>>> website and manual, ala Maven. Here the code in steps 1 and 2 will
>>>> be
>>>>>> reused.)
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers <
>>>> ralph.goers@dslextreme.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I wouldn’t necessarily want to penalize them. At the same time, I
>>>> don’t
>>>>>>> think we want a ton of extra repos each of which needs its own
>>>> release.
>>>>>>>
>>>>>>> Ralph
>>>>>>>
>>>>>>>> On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz <
>>>>>> piotr.karwasz@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <vo...@yazi.ci>
>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 with moving Cassandra et al. to their own repo
>>>>>>>>>
>>>>>>>>
>>>>>>>> I agree: Cassandra's tests were one of the most common reasons for
>>>>>>> failing
>>>>>>>> CI.
>>>>>>>> Maybe it could share a repository with other database appenders.
>>>>>>>>
>>>>>>>> Piotr
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: Upgradable dependencies
Posted by Ralph Goers <ra...@dslextreme.com>.
I believe there is a set of repos that does make sense. Volkan was on the right
track but got a few things incorrect, which I previously called out.
Ralph
> On May 1, 2022, at 1:56 PM, Gary Gregory <ga...@gmail.com> wrote:
>
> And yes, one repo per module is another option, a la Maven as you point
> out, but what is being proposed is some halfway solution which to me feels
> confusing and arbitrary just because today this and that module is in favor
> but that some others.
>
> Gaty
>
> On Sun, May 1, 2022, 07:54 Gary Gregory <ga...@gmail.com> wrote:
>
>> Hi Volkan,
>>
>> I see how you build and how you do version control as orthogonal concerns.
>> Currently we build and deliver everything from the root of the git checkout
>> directory. It does not have to be that way is all I'm saying. One could
>> just cd to a sub directory and work there. You could still have the option
>> to build the world from the root as a convenient way to make sure
>> everything works together and there is no breakage.
>>
>> https://en.m.wikipedia.org/wiki/Monorepo
>>
>> Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9] all
>> employ very large monorepos with varying strategies to scale build systems
>> and version control software with a large volume of code and daily changes.
>>
>> Log4j is tiny in comparison.
>>
>> Gary
>>
>> On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <vo...@yazi.ci> wrote:
>>
>>> I disagree that multi-repo setup is a recipe for disaster. It works
>>> perfectly well for Maven. Actually their setup is way more complicated:
>>> dedicated repo for each module! But let's assume you're right and
>>> multi-repos decrease the shelf time. Isn't that something good? I like
>>> repos that rot, it gives a clear message about that project: it simply is
>>> dead. If we migrate to multiple repos and some never get to make a
>>> release,
>>> what does this mean? Apparently nothing interesting is happening here. If
>>> there is a demand from the community to warrant a release, I am sure that
>>> release will come. After all, the very same community attempted to
>>> resurrect Log4j 1.
>>>
>>> I also like single repo setups where build and test times are within
>>> bearable limits, though Log4j is not one of them. Note that multi-repo
>>> discussion started, because we don't know any other way to solve the
>>> current set of problems. If you can think of how to address these concerns
>>> in some other way (e.g., "move folders around"), I am all ears.
>>>
>>> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <ga...@gmail.com>
>>> wrote:
>>>
>>>> Breaking our mono repo into a bunch of smaller repo pieces is a recipe
>>> for
>>>> software rot and orphanage IMO.
>>>>
>>>> For example, we have never released out of the tools repo, and I have
>>> folks
>>>> at work at use code from that repo for some internal facing work. Now, I
>>>> could take the time and do a release of course but... priorities. OTOH,
>>> I
>>>> could have gotten a release for "free" if that code was in the mono
>>> repo (I
>>>> am not advocating to put it back).
>>>>
>>>> This email might be better articulated in a call but I am trying to be
>>>> constructive: I think we can keep our mono repo and develop faster by
>>>> reorganizing what we have now. I like having everything in one place
>>>> myself. I am not sure how to actually move folders around to achieve
>>>> whatever new release cycle/split up we want, but I am sure it is do
>>> able.
>>>>
>>>> Gary
>>>>
>>>> On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <vo...@yazi.ci> wrote:
>>>>
>>>>> I have raised this issue in the past. I even came up with a proposal
>>> on
>>>> the
>>>>> repository structure
>>>>> <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3>
>>> to be
>>>>> followed. I am a big proponent of breaking up the logging-log4j2
>>>> repository
>>>>> into multiple ones. The current size of the code base is simply
>>>>> incomprehensible and slows down the development cycle tremendously
>>> due to
>>>>> excessive compilation and test times.
>>>>>
>>>>> *The proposal*
>>>>>
>>>>> Let me share the break down I have in mind:
>>>>>
>>>>> *logging-log4j2-api*
>>>>> log4j-1.2-api
>>>>> log4j-api
>>>>>
>>>>> *logging-log4j2*
>>>>> log4j-bom
>>>>> log4j-core
>>>>> log4j-core-its
>>>>> log4j-gctests
>>>>> log4j-iostreams
>>>>> log4j-perf
>>>>> log4j-plugins
>>>>>
>>>>> *logging-log4j2-appender*
>>>>> log4j-cassandra
>>>>> log4j-couchdb
>>>>> log4j-csv
>>>>> log4j-flume-ng
>>>>> log4j-mongodb3
>>>>> log4j-mongodb4
>>>>> log4j-smtp
>>>>>
>>>>> *logging-log4j2-appender-jdbc*
>>>>> log4j-jdbc (?)
>>>>> log4j-jdbc-dbcp2 (?)
>>>>> log4j-jpa
>>>>>
>>>>> *logging-log4j2-appender-queue*
>>>>> log4j-jeromq
>>>>> log4j-jms
>>>>> log4j-kafka
>>>>> log4j-redis
>>>>>
>>>>> *logging-log4j2-binding*
>>>>> log4j-jcl
>>>>> log4j-jpl
>>>>> log4j-jul
>>>>> log4j-liquibase
>>>>> log4j-slf4j18-impl
>>>>> log4j-slf4j-impl
>>>>> log4j-to-slf4j
>>>>>
>>>>> *logging-log4j2-container*
>>>>> log4j-docker
>>>>> log4j-kubernetes
>>>>>
>>>>> *logging-log4j2-gui*
>>>>> log4j-jmx-gui
>>>>>
>>>>> *logging-log4j2-jee*
>>>>> log4j-appserver
>>>>> log4j-taglib
>>>>> log4j-web
>>>>>
>>>>> *logging-log4j2-layout-json*
>>>>> log4j-layout-gelf
>>>>> log4j-layout-jackson
>>>>> log4j-layout-jackson-json
>>>>> log4j-layout-jackson-xml
>>>>> log4j-layout-jackson-yaml
>>>>> log4j-layout-template-json
>>>>>
>>>>> *logging-log4j2-osgi*
>>>>> log4j-osgi
>>>>>
>>>>> *logging-log4j2-spring*
>>>>> log4j-spring-boot
>>>>> log4j-spring-cloud-config
>>>>>
>>>>> Note that above breakdown contains every available module in master,
>>>> except
>>>>> log4j-samples, which has the following sub-modules:
>>>>>
>>>>> log4j-samples-loggerProperties (contains 2 very trivial example
>>> classes,
>>>>> log4j-core [tests] contains more comprehensive alternatives)
>>>>> log4j-samples-flume-remote (flume-specific)
>>>>> log4j-samples-flume-embedded (flume-specific)
>>>>> log4j-samples-flume-common (flume-specific)
>>>>> log4j-samples-configuration (contains 3 very trivial example classes,
>>>>> log4j-core [tests] contains more comprehensive alternatives)
>>>>>
>>>>> I propose removing the log4j-samples module and, if necessary, moving
>>>>> Flume-related parts to the log4j-flume-ng.
>>>>>
>>>>> *The problem with multiple repositories*
>>>>>
>>>>> I made this proposal in June 2021. Since then the more my familiarity
>>>> with
>>>>> the website and manual generation increased, the less I became
>>> inclined
>>>> to
>>>>> move forward. I am not much concerned about the increased release
>>> labor
>>>> the
>>>>> Ralph has mentioned – a majority of these repos will hardly get an
>>> update
>>>>> to warrant a release. Though the fact that current website generation
>>>>> depends on the presence of these modules to be in the same repository
>>> and
>>>>> with multiple repositories, the manual and source codes will be in
>>>>> different repositories... These concern me a lot. I think, the steps
>>> to
>>>>> overhaul the project structure should better be in the following
>>> order:
>>>>>
>>>>> 1. Replace `site` phase (`site` phase won't be needed anymore.
>>> Website
>>>>> will independently be generated in a single step via JBake.)
>>>>> 2. Rewrite the manual
>>>>> 3. Migrate to a multi-repo structure (Each repo will have its own
>>>>> website and manual, ala Maven. Here the code in steps 1 and 2 will
>>> be
>>>>> reused.)
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers <
>>> ralph.goers@dslextreme.com>
>>>>> wrote:
>>>>>
>>>>>> I wouldn’t necessarily want to penalize them. At the same time, I
>>> don’t
>>>>>> think we want a ton of extra repos each of which needs its own
>>> release.
>>>>>>
>>>>>> Ralph
>>>>>>
>>>>>>> On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz <
>>>>> piotr.karwasz@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <vo...@yazi.ci>
>>> wrote:
>>>>>>>
>>>>>>>> +1 with moving Cassandra et al. to their own repo
>>>>>>>>
>>>>>>>
>>>>>>> I agree: Cassandra's tests were one of the most common reasons for
>>>>>> failing
>>>>>>> CI.
>>>>>>> Maybe it could share a repository with other database appenders.
>>>>>>>
>>>>>>> Piotr
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: Upgradable dependencies
Posted by Gary Gregory <ga...@gmail.com>.
And yes, one repo per module is another option, a la Maven as you point
out, but what is being proposed is some halfway solution which to me feels
confusing and arbitrary just because today this and that module is in favor
but that some others.
Gaty
On Sun, May 1, 2022, 07:54 Gary Gregory <ga...@gmail.com> wrote:
> Hi Volkan,
>
> I see how you build and how you do version control as orthogonal concerns.
> Currently we build and deliver everything from the root of the git checkout
> directory. It does not have to be that way is all I'm saying. One could
> just cd to a sub directory and work there. You could still have the option
> to build the world from the root as a convenient way to make sure
> everything works together and there is no breakage.
>
> https://en.m.wikipedia.org/wiki/Monorepo
>
> Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9] all
> employ very large monorepos with varying strategies to scale build systems
> and version control software with a large volume of code and daily changes.
>
> Log4j is tiny in comparison.
>
> Gary
>
> On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <vo...@yazi.ci> wrote:
>
>> I disagree that multi-repo setup is a recipe for disaster. It works
>> perfectly well for Maven. Actually their setup is way more complicated:
>> dedicated repo for each module! But let's assume you're right and
>> multi-repos decrease the shelf time. Isn't that something good? I like
>> repos that rot, it gives a clear message about that project: it simply is
>> dead. If we migrate to multiple repos and some never get to make a
>> release,
>> what does this mean? Apparently nothing interesting is happening here. If
>> there is a demand from the community to warrant a release, I am sure that
>> release will come. After all, the very same community attempted to
>> resurrect Log4j 1.
>>
>> I also like single repo setups where build and test times are within
>> bearable limits, though Log4j is not one of them. Note that multi-repo
>> discussion started, because we don't know any other way to solve the
>> current set of problems. If you can think of how to address these concerns
>> in some other way (e.g., "move folders around"), I am all ears.
>>
>> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <ga...@gmail.com>
>> wrote:
>>
>> > Breaking our mono repo into a bunch of smaller repo pieces is a recipe
>> for
>> > software rot and orphanage IMO.
>> >
>> > For example, we have never released out of the tools repo, and I have
>> folks
>> > at work at use code from that repo for some internal facing work. Now, I
>> > could take the time and do a release of course but... priorities. OTOH,
>> I
>> > could have gotten a release for "free" if that code was in the mono
>> repo (I
>> > am not advocating to put it back).
>> >
>> > This email might be better articulated in a call but I am trying to be
>> > constructive: I think we can keep our mono repo and develop faster by
>> > reorganizing what we have now. I like having everything in one place
>> > myself. I am not sure how to actually move folders around to achieve
>> > whatever new release cycle/split up we want, but I am sure it is do
>> able.
>> >
>> > Gary
>> >
>> > On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <vo...@yazi.ci> wrote:
>> >
>> > > I have raised this issue in the past. I even came up with a proposal
>> on
>> > the
>> > > repository structure
>> > > <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3>
>> to be
>> > > followed. I am a big proponent of breaking up the logging-log4j2
>> > repository
>> > > into multiple ones. The current size of the code base is simply
>> > > incomprehensible and slows down the development cycle tremendously
>> due to
>> > > excessive compilation and test times.
>> > >
>> > > *The proposal*
>> > >
>> > > Let me share the break down I have in mind:
>> > >
>> > > *logging-log4j2-api*
>> > > log4j-1.2-api
>> > > log4j-api
>> > >
>> > > *logging-log4j2*
>> > > log4j-bom
>> > > log4j-core
>> > > log4j-core-its
>> > > log4j-gctests
>> > > log4j-iostreams
>> > > log4j-perf
>> > > log4j-plugins
>> > >
>> > > *logging-log4j2-appender*
>> > > log4j-cassandra
>> > > log4j-couchdb
>> > > log4j-csv
>> > > log4j-flume-ng
>> > > log4j-mongodb3
>> > > log4j-mongodb4
>> > > log4j-smtp
>> > >
>> > > *logging-log4j2-appender-jdbc*
>> > > log4j-jdbc (?)
>> > > log4j-jdbc-dbcp2 (?)
>> > > log4j-jpa
>> > >
>> > > *logging-log4j2-appender-queue*
>> > > log4j-jeromq
>> > > log4j-jms
>> > > log4j-kafka
>> > > log4j-redis
>> > >
>> > > *logging-log4j2-binding*
>> > > log4j-jcl
>> > > log4j-jpl
>> > > log4j-jul
>> > > log4j-liquibase
>> > > log4j-slf4j18-impl
>> > > log4j-slf4j-impl
>> > > log4j-to-slf4j
>> > >
>> > > *logging-log4j2-container*
>> > > log4j-docker
>> > > log4j-kubernetes
>> > >
>> > > *logging-log4j2-gui*
>> > > log4j-jmx-gui
>> > >
>> > > *logging-log4j2-jee*
>> > > log4j-appserver
>> > > log4j-taglib
>> > > log4j-web
>> > >
>> > > *logging-log4j2-layout-json*
>> > > log4j-layout-gelf
>> > > log4j-layout-jackson
>> > > log4j-layout-jackson-json
>> > > log4j-layout-jackson-xml
>> > > log4j-layout-jackson-yaml
>> > > log4j-layout-template-json
>> > >
>> > > *logging-log4j2-osgi*
>> > > log4j-osgi
>> > >
>> > > *logging-log4j2-spring*
>> > > log4j-spring-boot
>> > > log4j-spring-cloud-config
>> > >
>> > > Note that above breakdown contains every available module in master,
>> > except
>> > > log4j-samples, which has the following sub-modules:
>> > >
>> > > log4j-samples-loggerProperties (contains 2 very trivial example
>> classes,
>> > > log4j-core [tests] contains more comprehensive alternatives)
>> > > log4j-samples-flume-remote (flume-specific)
>> > > log4j-samples-flume-embedded (flume-specific)
>> > > log4j-samples-flume-common (flume-specific)
>> > > log4j-samples-configuration (contains 3 very trivial example classes,
>> > > log4j-core [tests] contains more comprehensive alternatives)
>> > >
>> > > I propose removing the log4j-samples module and, if necessary, moving
>> > > Flume-related parts to the log4j-flume-ng.
>> > >
>> > > *The problem with multiple repositories*
>> > >
>> > > I made this proposal in June 2021. Since then the more my familiarity
>> > with
>> > > the website and manual generation increased, the less I became
>> inclined
>> > to
>> > > move forward. I am not much concerned about the increased release
>> labor
>> > the
>> > > Ralph has mentioned – a majority of these repos will hardly get an
>> update
>> > > to warrant a release. Though the fact that current website generation
>> > > depends on the presence of these modules to be in the same repository
>> and
>> > > with multiple repositories, the manual and source codes will be in
>> > > different repositories... These concern me a lot. I think, the steps
>> to
>> > > overhaul the project structure should better be in the following
>> order:
>> > >
>> > > 1. Replace `site` phase (`site` phase won't be needed anymore.
>> Website
>> > > will independently be generated in a single step via JBake.)
>> > > 2. Rewrite the manual
>> > > 3. Migrate to a multi-repo structure (Each repo will have its own
>> > > website and manual, ala Maven. Here the code in steps 1 and 2 will
>> be
>> > > reused.)
>> > >
>> > >
>> > >
>> > > On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers <
>> ralph.goers@dslextreme.com>
>> > > wrote:
>> > >
>> > > > I wouldn’t necessarily want to penalize them. At the same time, I
>> don’t
>> > > > think we want a ton of extra repos each of which needs its own
>> release.
>> > > >
>> > > > Ralph
>> > > >
>> > > > > On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz <
>> > > piotr.karwasz@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > Hello,
>> > > > >
>> > > > > On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <vo...@yazi.ci>
>> wrote:
>> > > > >
>> > > > >> +1 with moving Cassandra et al. to their own repo
>> > > > >>
>> > > > >
>> > > > > I agree: Cassandra's tests were one of the most common reasons for
>> > > > failing
>> > > > > CI.
>> > > > > Maybe it could share a repository with other database appenders.
>> > > > >
>> > > > > Piotr
>> > > >
>> > > >
>> > >
>> >
>>
>