You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by Volkan Yazıcı <vo...@yazi.ci> on 2022/12/28 20:42:24 UTC
[log4j] The grand infra revamp
Hello,
I want to share some updates from my side on what I have been working on
for the last couple of months. In particular, I expect certain changes to
have an ASF-wide impact! Sounds interesting? Continue reading.
*What is up with `maven-changes-plugin`?*
The current `site` phase takes ages to complete. This significantly hinders
releases and more frequent website updates. The main culprit is `site`
phase plugins need to be executed for 50+ modules. As my investigation
turned out, we can actually either drop or replace all such plugins, except
one: `maven-changes-plugin`, that is used to generate the changelog from
`changes.xml`. `maven-changes-plugin` is also a major source of
merge-conflicts in between feature branches and PRs, since they all need to
touch to the same file: `changes.xml`.
*Is it only about overhauling the `site` phase?*
No. As we agreed on, we want to migrate the issue tracking from JIRA to
GitHub Issues. `maven-changes-plugin` blocks us there too, since it only
supports a single issue tracking system.
*Enter `log4j-changelog`*
I have written simple Java command line applications to perform the
following tasks:
1. Populate
`src/site/changelog/<releaseVersion>/<issueId>_<description>.xml` files
from `changes.xml`
2. Generate AsciiDoc-formatted changelog files from the populated
`src/site/changelog`
3. Make `src/site/changelog` ready before a release
These technical tools will not only help us to replace
`maven-changes-plugin`, support multiple issue trackers, and enable great
simplification of the `site` phase, due to its one-changelog-file-per-issue
convention, they will make changelog merge conflicts a thing of the past
too!
Okay, great! Since the `maven-changes-plugin` successor is there, what are
we exactly waiting for?
*Enter `log4j-tools`*
I have implemented `log4j-changelog` as a not-deployed Log4j module, though
Ralph insisted on not having this within Log4j to make it available for
other Log4j projects and since it needs to be shared by (and hence, sync'ed
in between) `release-2.x` and `master` branches. He proposed that we host
this in `log4j-tools` <https://github.com/apache/logging-log4j-tools>.
Since `log4j-tools` repository never had a release before, I let my
imagination go wild there:
- Adopted the Maven-recommended way of setting up a BOM:
`logging-parent` parents `log4j-bom`, which parents `log4j-parent`, which
parents all other modules. There the effective POM of `log4j-bom` is
stripped-down from its parent and all unnecessary weight via
`flatten-maven-plugin` voodoo.
- Switched to CI-friendly Maven versioning
<https://maven.apache.org/maven-ci-friendly.html>
- Configured GitHub Actions to make releases!
(As you might guess, if experiment succeeds, I will port all these fancy
stuff to Log4j too.)
*Releasing... via GitHub Actions!?!*
ASF project artifacts are required to be signed and up until now that has
**always** been performed manually by release maintainers. Releasing **and
signing** artifacts in a fully-automated fashion in CI... "Surely You're
Joking, Mr. Feynman!" But I am not! I have shared a technical
implementation proposal in `members@`, got approval from ASF Security Team
head Mark J. Cox
<https://lists.apache.org/thread/t1fbn11m70sy9df86xgzzp0fllg38p9q>, and now
I am waiting for INFRA-23996
<https://issues.apache.org/jira/browse/INFRA-23996> to be implemented!
What does this effectively mean? To the best of my knowledge, *`log4j-tools`
will be the first ASF project ever that is released via CI!* This will
enable various release automation enhancements, not to mention the software
supply chain traceability it will bring as well.
*Where are we now?*
Once INFRA-23996 <https://issues.apache.org/jira/browse/INFRA-23996> is
implemented and `log4j-tools` (containing `log4j-changelog`) is released,
the following PR train will be merged:
- #1145 <https://github.com/apache/logging-log4j2/pull/1145> – replace
`maven-changes-plugin` with `log4j-changelog`
- #1166 <https://github.com/apache/logging-log4j2/pull/1166> – simplify
`site` goal (`./mvnw site` takes ~30 seconds!)
- #1172 <https://github.com/apache/logging-log4j2/pull/1172> – update
sources, docs, links, etc. to reflect the migration from JIRA to GitHub
Issues
*What are we waiting for?*
We *_can_* release `log4j-tools` without INFRA-23996
<https://issues.apache.org/jira/browse/INFRA-23996>, the old school way,
i.e., via running `./mvnw deploy` from a personal laptop. That said, I
believe releasing `log4j-tools` via CI will have a way bigger impact beyond
the scope of Log4j, and `log4j-tools` constitutes a great candidate for
such an experiment. We are all set: `log4j-tools` GitHub Actions workflows
<https://github.com/apache/logging-log4j-tools/blob/master/.github/workflows/build.yml>
are ready to make an automated release. Infra team has indicated in
INFRA-23996 <https://issues.apache.org/jira/browse/INFRA-23996> that they
will implement it in a couple of days. Given how close we are to an
automated release, I will wait for my grand infra revamp scheme to unfold.
Re: [log4j] The grand infra revamp
Posted by Volkan Yazıcı <vo...@yazi.ci>.
Daniel Gruno stated in INFRA-23996 that they will implement the ticket in a
couple of days, I will wait for a week.
You can go ahead and merge your changes with necessary `changes.xml`
modifications, just like what we have done before. I will take care of
resolving `changes.xml`-related merge conflicts myself.
On Thu, Dec 29, 2022 at 8:21 AM Ralph Goers <ra...@dslextreme.com>
wrote:
> Volkan,
>
> Please do not wait to release log4j-tools. I am already holding off
> adding more changelog entries so as to avoid creating more work when the PR
> is merged. As far as I can tell there is no requirement that log4j-tools
> MUST be released via CI.
>
> I really would have liked to have all of this done already as this would
> have been an ideal time to release 2.19.1.
>
> Ralph
>
> > On Dec 28, 2022, at 6:34 PM, Matt Sicker <ma...@musigma.org> wrote:
> >
> > This sounds amazing! Thanks for championing this effort! For signing,
> there’s some project out there that lets you use OIDC for signing things
> and keeping a sort of certificate transparency log so that you can verify
> that not only was an artifact signed by a valid signature, but it’s also
> done by the expected key for a particular release. See
> https://www.sigstore.dev/ for details. I’m not sure if it works with GPG
> keys or Maven or anything, but it could be a useful basis for further
> release automation in the future (even if it’s something more custom to the
> ASF).
> > —
> > Matt Sicker
> >
> >> On Dec 28, 2022, at 14:42, Volkan Yazıcı <vo...@yazi.ci> wrote:
> >>
> >> Hello,
> >>
> >> I want to share some updates from my side on what I have been working on
> >> for the last couple of months. In particular, I expect certain changes
> to
> >> have an ASF-wide impact! Sounds interesting? Continue reading.
> >>
> >> *What is up with `maven-changes-plugin`?*
> >>
> >> The current `site` phase takes ages to complete. This significantly
> hinders
> >> releases and more frequent website updates. The main culprit is `site`
> >> phase plugins need to be executed for 50+ modules. As my investigation
> >> turned out, we can actually either drop or replace all such plugins,
> except
> >> one: `maven-changes-plugin`, that is used to generate the changelog from
> >> `changes.xml`. `maven-changes-plugin` is also a major source of
> >> merge-conflicts in between feature branches and PRs, since they all
> need to
> >> touch to the same file: `changes.xml`.
> >>
> >> *Is it only about overhauling the `site` phase?*
> >>
> >> No. As we agreed on, we want to migrate the issue tracking from JIRA to
> >> GitHub Issues. `maven-changes-plugin` blocks us there too, since it only
> >> supports a single issue tracking system.
> >>
> >> *Enter `log4j-changelog`*
> >>
> >> I have written simple Java command line applications to perform the
> >> following tasks:
> >>
> >> 1. Populate
> >> `src/site/changelog/<releaseVersion>/<issueId>_<description>.xml` files
> >> from `changes.xml`
> >> 2. Generate AsciiDoc-formatted changelog files from the populated
> >> `src/site/changelog`
> >> 3. Make `src/site/changelog` ready before a release
> >>
> >> These technical tools will not only help us to replace
> >> `maven-changes-plugin`, support multiple issue trackers, and enable
> great
> >> simplification of the `site` phase, due to its
> one-changelog-file-per-issue
> >> convention, they will make changelog merge conflicts a thing of the past
> >> too!
> >>
> >> Okay, great! Since the `maven-changes-plugin` successor is there, what
> are
> >> we exactly waiting for?
> >>
> >> *Enter `log4j-tools`*
> >>
> >> I have implemented `log4j-changelog` as a not-deployed Log4j module,
> though
> >> Ralph insisted on not having this within Log4j to make it available for
> >> other Log4j projects and since it needs to be shared by (and hence,
> sync'ed
> >> in between) `release-2.x` and `master` branches. He proposed that we
> host
> >> this in `log4j-tools` <https://github.com/apache/logging-log4j-tools>.
> >> Since `log4j-tools` repository never had a release before, I let my
> >> imagination go wild there:
> >>
> >> - Adopted the Maven-recommended way of setting up a BOM:
> >> `logging-parent` parents `log4j-bom`, which parents `log4j-parent`,
> which
> >> parents all other modules. There the effective POM of `log4j-bom` is
> >> stripped-down from its parent and all unnecessary weight via
> >> `flatten-maven-plugin` voodoo.
> >> - Switched to CI-friendly Maven versioning
> >> <https://maven.apache.org/maven-ci-friendly.html>
> >> - Configured GitHub Actions to make releases!
> >>
> >> (As you might guess, if experiment succeeds, I will port all these fancy
> >> stuff to Log4j too.)
> >>
> >> *Releasing... via GitHub Actions!?!*
> >>
> >> ASF project artifacts are required to be signed and up until now that
> has
> >> **always** been performed manually by release maintainers. Releasing
> **and
> >> signing** artifacts in a fully-automated fashion in CI... "Surely You're
> >> Joking, Mr. Feynman!" But I am not! I have shared a technical
> >> implementation proposal in `members@`, got approval from ASF Security
> Team
> >> head Mark J. Cox
> >> <https://lists.apache.org/thread/t1fbn11m70sy9df86xgzzp0fllg38p9q>,
> and now
> >> I am waiting for INFRA-23996
> >> <https://issues.apache.org/jira/browse/INFRA-23996> to be implemented!
> >>
> >> What does this effectively mean? To the best of my knowledge,
> *`log4j-tools`
> >> will be the first ASF project ever that is released via CI!* This will
> >> enable various release automation enhancements, not to mention the
> software
> >> supply chain traceability it will bring as well.
> >>
> >> *Where are we now?*
> >>
> >> Once INFRA-23996 <https://issues.apache.org/jira/browse/INFRA-23996> is
> >> implemented and `log4j-tools` (containing `log4j-changelog`) is
> released,
> >> the following PR train will be merged:
> >>
> >> - #1145 <https://github.com/apache/logging-log4j2/pull/1145> – replace
> >> `maven-changes-plugin` with `log4j-changelog`
> >> - #1166 <https://github.com/apache/logging-log4j2/pull/1166> –
> simplify
> >> `site` goal (`./mvnw site` takes ~30 seconds!)
> >> - #1172 <https://github.com/apache/logging-log4j2/pull/1172> – update
> >> sources, docs, links, etc. to reflect the migration from JIRA to GitHub
> >> Issues
> >>
> >> *What are we waiting for?*
> >>
> >> We *_can_* release `log4j-tools` without INFRA-23996
> >> <https://issues.apache.org/jira/browse/INFRA-23996>, the old school
> way,
> >> i.e., via running `./mvnw deploy` from a personal laptop. That said, I
> >> believe releasing `log4j-tools` via CI will have a way bigger impact
> beyond
> >> the scope of Log4j, and `log4j-tools` constitutes a great candidate for
> >> such an experiment. We are all set: `log4j-tools` GitHub Actions
> workflows
> >> <
> https://github.com/apache/logging-log4j-tools/blob/master/.github/workflows/build.yml
> >
> >> are ready to make an automated release. Infra team has indicated in
> >> INFRA-23996 <https://issues.apache.org/jira/browse/INFRA-23996> that
> they
> >> will implement it in a couple of days. Given how close we are to an
> >> automated release, I will wait for my grand infra revamp scheme to
> unfold.
> >
>
>
Re: [log4j] The grand infra revamp
Posted by Ralph Goers <ra...@dslextreme.com>.
Volkan,
Please do not wait to release log4j-tools. I am already holding off adding more changelog entries so as to avoid creating more work when the PR is merged. As far as I can tell there is no requirement that log4j-tools MUST be released via CI.
I really would have liked to have all of this done already as this would have been an ideal time to release 2.19.1.
Ralph
> On Dec 28, 2022, at 6:34 PM, Matt Sicker <ma...@musigma.org> wrote:
>
> This sounds amazing! Thanks for championing this effort! For signing, there’s some project out there that lets you use OIDC for signing things and keeping a sort of certificate transparency log so that you can verify that not only was an artifact signed by a valid signature, but it’s also done by the expected key for a particular release. See https://www.sigstore.dev/ for details. I’m not sure if it works with GPG keys or Maven or anything, but it could be a useful basis for further release automation in the future (even if it’s something more custom to the ASF).
> —
> Matt Sicker
>
>> On Dec 28, 2022, at 14:42, Volkan Yazıcı <vo...@yazi.ci> wrote:
>>
>> Hello,
>>
>> I want to share some updates from my side on what I have been working on
>> for the last couple of months. In particular, I expect certain changes to
>> have an ASF-wide impact! Sounds interesting? Continue reading.
>>
>> *What is up with `maven-changes-plugin`?*
>>
>> The current `site` phase takes ages to complete. This significantly hinders
>> releases and more frequent website updates. The main culprit is `site`
>> phase plugins need to be executed for 50+ modules. As my investigation
>> turned out, we can actually either drop or replace all such plugins, except
>> one: `maven-changes-plugin`, that is used to generate the changelog from
>> `changes.xml`. `maven-changes-plugin` is also a major source of
>> merge-conflicts in between feature branches and PRs, since they all need to
>> touch to the same file: `changes.xml`.
>>
>> *Is it only about overhauling the `site` phase?*
>>
>> No. As we agreed on, we want to migrate the issue tracking from JIRA to
>> GitHub Issues. `maven-changes-plugin` blocks us there too, since it only
>> supports a single issue tracking system.
>>
>> *Enter `log4j-changelog`*
>>
>> I have written simple Java command line applications to perform the
>> following tasks:
>>
>> 1. Populate
>> `src/site/changelog/<releaseVersion>/<issueId>_<description>.xml` files
>> from `changes.xml`
>> 2. Generate AsciiDoc-formatted changelog files from the populated
>> `src/site/changelog`
>> 3. Make `src/site/changelog` ready before a release
>>
>> These technical tools will not only help us to replace
>> `maven-changes-plugin`, support multiple issue trackers, and enable great
>> simplification of the `site` phase, due to its one-changelog-file-per-issue
>> convention, they will make changelog merge conflicts a thing of the past
>> too!
>>
>> Okay, great! Since the `maven-changes-plugin` successor is there, what are
>> we exactly waiting for?
>>
>> *Enter `log4j-tools`*
>>
>> I have implemented `log4j-changelog` as a not-deployed Log4j module, though
>> Ralph insisted on not having this within Log4j to make it available for
>> other Log4j projects and since it needs to be shared by (and hence, sync'ed
>> in between) `release-2.x` and `master` branches. He proposed that we host
>> this in `log4j-tools` <https://github.com/apache/logging-log4j-tools>.
>> Since `log4j-tools` repository never had a release before, I let my
>> imagination go wild there:
>>
>> - Adopted the Maven-recommended way of setting up a BOM:
>> `logging-parent` parents `log4j-bom`, which parents `log4j-parent`, which
>> parents all other modules. There the effective POM of `log4j-bom` is
>> stripped-down from its parent and all unnecessary weight via
>> `flatten-maven-plugin` voodoo.
>> - Switched to CI-friendly Maven versioning
>> <https://maven.apache.org/maven-ci-friendly.html>
>> - Configured GitHub Actions to make releases!
>>
>> (As you might guess, if experiment succeeds, I will port all these fancy
>> stuff to Log4j too.)
>>
>> *Releasing... via GitHub Actions!?!*
>>
>> ASF project artifacts are required to be signed and up until now that has
>> **always** been performed manually by release maintainers. Releasing **and
>> signing** artifacts in a fully-automated fashion in CI... "Surely You're
>> Joking, Mr. Feynman!" But I am not! I have shared a technical
>> implementation proposal in `members@`, got approval from ASF Security Team
>> head Mark J. Cox
>> <https://lists.apache.org/thread/t1fbn11m70sy9df86xgzzp0fllg38p9q>, and now
>> I am waiting for INFRA-23996
>> <https://issues.apache.org/jira/browse/INFRA-23996> to be implemented!
>>
>> What does this effectively mean? To the best of my knowledge, *`log4j-tools`
>> will be the first ASF project ever that is released via CI!* This will
>> enable various release automation enhancements, not to mention the software
>> supply chain traceability it will bring as well.
>>
>> *Where are we now?*
>>
>> Once INFRA-23996 <https://issues.apache.org/jira/browse/INFRA-23996> is
>> implemented and `log4j-tools` (containing `log4j-changelog`) is released,
>> the following PR train will be merged:
>>
>> - #1145 <https://github.com/apache/logging-log4j2/pull/1145> – replace
>> `maven-changes-plugin` with `log4j-changelog`
>> - #1166 <https://github.com/apache/logging-log4j2/pull/1166> – simplify
>> `site` goal (`./mvnw site` takes ~30 seconds!)
>> - #1172 <https://github.com/apache/logging-log4j2/pull/1172> – update
>> sources, docs, links, etc. to reflect the migration from JIRA to GitHub
>> Issues
>>
>> *What are we waiting for?*
>>
>> We *_can_* release `log4j-tools` without INFRA-23996
>> <https://issues.apache.org/jira/browse/INFRA-23996>, the old school way,
>> i.e., via running `./mvnw deploy` from a personal laptop. That said, I
>> believe releasing `log4j-tools` via CI will have a way bigger impact beyond
>> the scope of Log4j, and `log4j-tools` constitutes a great candidate for
>> such an experiment. We are all set: `log4j-tools` GitHub Actions workflows
>> <https://github.com/apache/logging-log4j-tools/blob/master/.github/workflows/build.yml>
>> are ready to make an automated release. Infra team has indicated in
>> INFRA-23996 <https://issues.apache.org/jira/browse/INFRA-23996> that they
>> will implement it in a couple of days. Given how close we are to an
>> automated release, I will wait for my grand infra revamp scheme to unfold.
>
Re: [log4j] The grand infra revamp
Posted by Matt Sicker <ma...@musigma.org>.
This sounds amazing! Thanks for championing this effort! For signing, there’s some project out there that lets you use OIDC for signing things and keeping a sort of certificate transparency log so that you can verify that not only was an artifact signed by a valid signature, but it’s also done by the expected key for a particular release. See https://www.sigstore.dev/ for details. I’m not sure if it works with GPG keys or Maven or anything, but it could be a useful basis for further release automation in the future (even if it’s something more custom to the ASF).
—
Matt Sicker
> On Dec 28, 2022, at 14:42, Volkan Yazıcı <vo...@yazi.ci> wrote:
>
> Hello,
>
> I want to share some updates from my side on what I have been working on
> for the last couple of months. In particular, I expect certain changes to
> have an ASF-wide impact! Sounds interesting? Continue reading.
>
> *What is up with `maven-changes-plugin`?*
>
> The current `site` phase takes ages to complete. This significantly hinders
> releases and more frequent website updates. The main culprit is `site`
> phase plugins need to be executed for 50+ modules. As my investigation
> turned out, we can actually either drop or replace all such plugins, except
> one: `maven-changes-plugin`, that is used to generate the changelog from
> `changes.xml`. `maven-changes-plugin` is also a major source of
> merge-conflicts in between feature branches and PRs, since they all need to
> touch to the same file: `changes.xml`.
>
> *Is it only about overhauling the `site` phase?*
>
> No. As we agreed on, we want to migrate the issue tracking from JIRA to
> GitHub Issues. `maven-changes-plugin` blocks us there too, since it only
> supports a single issue tracking system.
>
> *Enter `log4j-changelog`*
>
> I have written simple Java command line applications to perform the
> following tasks:
>
> 1. Populate
> `src/site/changelog/<releaseVersion>/<issueId>_<description>.xml` files
> from `changes.xml`
> 2. Generate AsciiDoc-formatted changelog files from the populated
> `src/site/changelog`
> 3. Make `src/site/changelog` ready before a release
>
> These technical tools will not only help us to replace
> `maven-changes-plugin`, support multiple issue trackers, and enable great
> simplification of the `site` phase, due to its one-changelog-file-per-issue
> convention, they will make changelog merge conflicts a thing of the past
> too!
>
> Okay, great! Since the `maven-changes-plugin` successor is there, what are
> we exactly waiting for?
>
> *Enter `log4j-tools`*
>
> I have implemented `log4j-changelog` as a not-deployed Log4j module, though
> Ralph insisted on not having this within Log4j to make it available for
> other Log4j projects and since it needs to be shared by (and hence, sync'ed
> in between) `release-2.x` and `master` branches. He proposed that we host
> this in `log4j-tools` <https://github.com/apache/logging-log4j-tools>.
> Since `log4j-tools` repository never had a release before, I let my
> imagination go wild there:
>
> - Adopted the Maven-recommended way of setting up a BOM:
> `logging-parent` parents `log4j-bom`, which parents `log4j-parent`, which
> parents all other modules. There the effective POM of `log4j-bom` is
> stripped-down from its parent and all unnecessary weight via
> `flatten-maven-plugin` voodoo.
> - Switched to CI-friendly Maven versioning
> <https://maven.apache.org/maven-ci-friendly.html>
> - Configured GitHub Actions to make releases!
>
> (As you might guess, if experiment succeeds, I will port all these fancy
> stuff to Log4j too.)
>
> *Releasing... via GitHub Actions!?!*
>
> ASF project artifacts are required to be signed and up until now that has
> **always** been performed manually by release maintainers. Releasing **and
> signing** artifacts in a fully-automated fashion in CI... "Surely You're
> Joking, Mr. Feynman!" But I am not! I have shared a technical
> implementation proposal in `members@`, got approval from ASF Security Team
> head Mark J. Cox
> <https://lists.apache.org/thread/t1fbn11m70sy9df86xgzzp0fllg38p9q>, and now
> I am waiting for INFRA-23996
> <https://issues.apache.org/jira/browse/INFRA-23996> to be implemented!
>
> What does this effectively mean? To the best of my knowledge, *`log4j-tools`
> will be the first ASF project ever that is released via CI!* This will
> enable various release automation enhancements, not to mention the software
> supply chain traceability it will bring as well.
>
> *Where are we now?*
>
> Once INFRA-23996 <https://issues.apache.org/jira/browse/INFRA-23996> is
> implemented and `log4j-tools` (containing `log4j-changelog`) is released,
> the following PR train will be merged:
>
> - #1145 <https://github.com/apache/logging-log4j2/pull/1145> – replace
> `maven-changes-plugin` with `log4j-changelog`
> - #1166 <https://github.com/apache/logging-log4j2/pull/1166> – simplify
> `site` goal (`./mvnw site` takes ~30 seconds!)
> - #1172 <https://github.com/apache/logging-log4j2/pull/1172> – update
> sources, docs, links, etc. to reflect the migration from JIRA to GitHub
> Issues
>
> *What are we waiting for?*
>
> We *_can_* release `log4j-tools` without INFRA-23996
> <https://issues.apache.org/jira/browse/INFRA-23996>, the old school way,
> i.e., via running `./mvnw deploy` from a personal laptop. That said, I
> believe releasing `log4j-tools` via CI will have a way bigger impact beyond
> the scope of Log4j, and `log4j-tools` constitutes a great candidate for
> such an experiment. We are all set: `log4j-tools` GitHub Actions workflows
> <https://github.com/apache/logging-log4j-tools/blob/master/.github/workflows/build.yml>
> are ready to make an automated release. Infra team has indicated in
> INFRA-23996 <https://issues.apache.org/jira/browse/INFRA-23996> that they
> will implement it in a couple of days. Given how close we are to an
> automated release, I will wait for my grand infra revamp scheme to unfold.