You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by "Piotr P. Karwasz" <pi...@mailing.copernik.eu> on 2025/01/19 09:02:52 UTC
Create artifacts for optional features
Hi all,
Maven is notably bad at handling optional dependencies, which are
basically ignored by consumers of the POM file. While Log4j Core 3
solves this problem by not having **any** optional dependencies, we
could backport this feature also to Log4j Core 2.
What do you think about adding empty artifacts in Log4j Core 2 for each
optional feature? They would work exactly like Spring Boot starters and
pull the required modules and dependencies into a user's project. For
example `log4j-async-logger` would depend on `com.lmax:disruptor`.
The purpose of such artifacts would be double:
* They would allow a smoother migration to Log4j Core 3 (just a version
bump). An empty `log4j-async-logger` from version 2.x, will be replaced
with a real JAR in version 3.x.
* They would a semantic purpose: they would allow developers better
understand, why a certain dependency is in their stack. This is
particularly important in case of vulnerabilities in the dependencies.
For example, if the user has `log4j-config-yaml` as dependency and a CVE
is published against SnakeYaml, he can easily deduce that the CVE would
only affect Log4j Core. Even for an empty `log4j-config-yaml` we can
publish a VEX statement that says Log4j Core is not affected, since it
does not parse YAML files from untrusted sources. This scenario was
discussed on `dev@kafka`[1].
I think we could backport from Log4j Core 3 as empty/starter JARs the
following modules:
* `log4j-async-logger` for the Asynchronous Loggers feature.
* `log4j-config-properties` and `log4j-config-yaml`. In the future we
might add `log4j-config-xml` in Log4j Core 3 as empty JAR, but with a
`requires java.xml` module descriptor. I am not sure if it is worth
introducing a `log4j-config-json` in version 2.x, since version 3.x does
not require any additional deps for JSON.
* `log4j-compress` for alternative compression algorithms of the rolling
file appenders.
* `log4j-conversant` and `log4j-jctools` for alternative queue factories
of the Asynchronous Appender.
* `log4j-csv` for the CSV layout.
* `log4j-jndi` for the optional dependency on `java.naming`.
* `log4j-jdbc` and `log4j-jdbc-jndi` for the JDBC Appender and its JNDI
connection source.
* `log4j-script` for the optional dependency on `java.scripting`.
Piotr
[1] https://lists.apache.org/thread/bnrr6djsd7nf2135ycpb9xmkwr7c8pt7
Re: Create artifacts for optional features
Posted by Matt Sicker <ma...@musigma.org>.
This is essentially what I proposed long ago in Log4j Boot. They’d be pom files with the appropriate dependencies marked in for simplicity of end users. Could still be useful.
> On Jan 19, 2025, at 03:02, Piotr P. Karwasz <pi...@mailing.copernik.eu> wrote:
>
> Hi all,
>
> Maven is notably bad at handling optional dependencies, which are basically ignored by consumers of the POM file. While Log4j Core 3 solves this problem by not having **any** optional dependencies, we could backport this feature also to Log4j Core 2.
>
> What do you think about adding empty artifacts in Log4j Core 2 for each optional feature? They would work exactly like Spring Boot starters and pull the required modules and dependencies into a user's project. For example `log4j-async-logger` would depend on `com.lmax:disruptor`.
>
> The purpose of such artifacts would be double:
>
> * They would allow a smoother migration to Log4j Core 3 (just a version bump). An empty `log4j-async-logger` from version 2.x, will be replaced with a real JAR in version 3.x.
>
> * They would a semantic purpose: they would allow developers better understand, why a certain dependency is in their stack. This is particularly important in case of vulnerabilities in the dependencies. For example, if the user has `log4j-config-yaml` as dependency and a CVE is published against SnakeYaml, he can easily deduce that the CVE would only affect Log4j Core. Even for an empty `log4j-config-yaml` we can publish a VEX statement that says Log4j Core is not affected, since it does not parse YAML files from untrusted sources. This scenario was discussed on `dev@kafka`[1].
>
> I think we could backport from Log4j Core 3 as empty/starter JARs the following modules:
>
> * `log4j-async-logger` for the Asynchronous Loggers feature.
>
> * `log4j-config-properties` and `log4j-config-yaml`. In the future we might add `log4j-config-xml` in Log4j Core 3 as empty JAR, but with a `requires java.xml` module descriptor. I am not sure if it is worth introducing a `log4j-config-json` in version 2.x, since version 3.x does not require any additional deps for JSON.
>
> * `log4j-compress` for alternative compression algorithms of the rolling file appenders.
>
> * `log4j-conversant` and `log4j-jctools` for alternative queue factories of the Asynchronous Appender.
>
> * `log4j-csv` for the CSV layout.
>
> * `log4j-jndi` for the optional dependency on `java.naming`.
>
> * `log4j-jdbc` and `log4j-jdbc-jndi` for the JDBC Appender and its JNDI connection source.
>
> * `log4j-script` for the optional dependency on `java.scripting`.
>
> Piotr
>
> [1] https://lists.apache.org/thread/bnrr6djsd7nf2135ycpb9xmkwr7c8pt7
>
Re: Create artifacts for optional features
Posted by "Piotr P. Karwasz" <pi...@mailing.copernik.eu>.
Hi Gary,
On 21.01.2025 13:46, Gary Gregory wrote:
> This all sounds very confusing to me. Using this scheme, I could
> create a set of dependencies on 2.x with empty jars that don't work on
> 3.x (because I picked the wrong jars). There is no way to verify that
> what I'm doing on 2.x will work on 3.x without trying it on 3.x, so...
> or am I missing something?
The list of optional features in Log4j 2 is quite limited and
documented, I don't think there is a way to add the wrong dependencies.
Basically we have two choices:
* Tell users that they need to add `jackson-dataformat-yaml` to use the
YAML configuration format. This is what we do today.
* Tell users that they need to add `log4j-config-yaml` to use the YAML
configuration format and make it depend on `jackson-dataformat-yaml`. If
they only add `jackson-dataformat-yaml`, it will still work in 2.x.
The first option only works with Log4j 2, the second one would work with
Log4j 2 and 3. That is all there is.
Piotr
[1] https://logging.apache.org/log4j/3.x/migrate-from-log4j2.html
Re: Create artifacts for optional features
Posted by Gary Gregory <ga...@gmail.com>.
This all sounds very confusing to me. Using this scheme, I could
create a set of dependencies on 2.x with empty jars that don't work on
3.x (because I picked the wrong jars). There is no way to verify that
what I'm doing on 2.x will work on 3.x without trying it on 3.x, so...
or am I missing something?
Gary
On Sun, Jan 19, 2025 at 1:32 PM Piotr P. Karwasz
<pi...@mailing.copernik.eu> wrote:
>
> Hi Robert,
>
> On 19.01.2025 18:39, Robert Middleton wrote:
> > If I'm understanding this correctly, this sounds like a bad idea.
> > While it does make the 2.x -> 3.x transition easier, it creates a
> > bigger issue for when you go from 2.x -> 2.y.
>
> As long as `x` is higher than `y` there will be no problems: the
> additional artifacts are not necessary in 2.x, they are just semantic
> sugar. Users can still add the third-party dependencies like
> `com.lmax:disruptor` themselves, but they could also just add
> `log4j-async-logger` and let us choose the appropriate `disruptor`
> version to use.
>
> The additional artifacts are also meant to help solving security alerts.
> Imagine a CVE is published against XZ that affects the decompression of
> archives. The developer of Acme Cool Server:
>
> 1. Finds out that `xz` is a transitive dependency of his app.
>
> 2. Finds out that `xz` is pulled by the direct dependency
> `commons-compress`.
>
> 3. Asks the Commons project if the problem in `xz` can affect
> `commons-compress`.
>
> 4. The answer will probably be "if you use decompression, YES" and he
> will need to dive into the code to see how Commons Compress is used.
>
> 5. He won't find anything in its own code and he will spend several
> hours to find out the Commons Compress is used by `log4j-core`.
>
> 6. Asks us if the problem in `xz` can affect `log4j-core`.
>
> 7. We will answer "NO, the affected code is not reachable".
>
> If we introduce a `log4j-compress` artifact that depends on
> `commons-compress` and the application uses it, steps 3-5 can be skipped.
>
> Piotr
>
Re: Create artifacts for optional features
Posted by "Piotr P. Karwasz" <pi...@mailing.copernik.eu>.
Hi Robert,
On 19.01.2025 18:39, Robert Middleton wrote:
> If I'm understanding this correctly, this sounds like a bad idea.
> While it does make the 2.x -> 3.x transition easier, it creates a
> bigger issue for when you go from 2.x -> 2.y.
As long as `x` is higher than `y` there will be no problems: the
additional artifacts are not necessary in 2.x, they are just semantic
sugar. Users can still add the third-party dependencies like
`com.lmax:disruptor` themselves, but they could also just add
`log4j-async-logger` and let us choose the appropriate `disruptor`
version to use.
The additional artifacts are also meant to help solving security alerts.
Imagine a CVE is published against XZ that affects the decompression of
archives. The developer of Acme Cool Server:
1. Finds out that `xz` is a transitive dependency of his app.
2. Finds out that `xz` is pulled by the direct dependency
`commons-compress`.
3. Asks the Commons project if the problem in `xz` can affect
`commons-compress`.
4. The answer will probably be "if you use decompression, YES" and he
will need to dive into the code to see how Commons Compress is used.
5. He won't find anything in its own code and he will spend several
hours to find out the Commons Compress is used by `log4j-core`.
6. Asks us if the problem in `xz` can affect `log4j-core`.
7. We will answer "NO, the affected code is not reachable".
If we introduce a `log4j-compress` artifact that depends on
`commons-compress` and the application uses it, steps 3-5 can be skipped.
Piotr
Re: Create artifacts for optional features
Posted by Robert Middleton <rm...@apache.org>.
If I'm understanding this correctly, this sounds like a bad idea.
While it does make the 2.x -> 3.x transition easier, it creates a
bigger issue for when you go from 2.x -> 2.y. Personally I always
find it very annoying when software makes big changes like this on a
minor release, since it certainly goes against the spirit of semantic
versioning.
This change is better documented in a "migrate from 2.x to 3.x" page.
-Robert Middleton
On Sun, Jan 19, 2025 at 12:17 PM Piotr P. Karwasz
<pi...@mailing.copernik.eu> wrote:
>
> Hi Ralph,
>
> On 19.01.2025 17:55, Ralph Goers wrote:
> > I must be misunderstanding. IIUC you are proposing that to eliminate optional dependencies we create a bunch of empty jars, presumably with a pom that has the dependency as required. How exactly does that help? Doesn’t that new, empty artifact just become an optional dependency? Isn’t it required during the build to allow the dependency to be resolved?
>
> As you remark, the optional dependencies in `log4j-core` must remain (at
> least at compile time, so they must be either `provided` or
> `compile+optional` like now). What I am proposing is to create a new
> `log4j-async-logger` artifact with a non-optional dependency on
> `com.lmax:disruptor`. We should then change the installation
> documentation for async loggers[1] and ask users to add:
>
>
> <dependency>
>
> <groupId>org.apache.logging.log4j</groupId>
>
> <artifactId>log4j-async-logger</artifactId>
>
> <scope>runtime</scope>
>
> </dependency>
>
>
> The advantage of such a change is that:
>
> * It makes the upgrade to Log4j Core 3 trivial (the same dependency is
> required in Log4j Core 3[2])
>
> * Users no longer have to remember what `disruptor` is and why they
> added it to their application. They will upgrade the version of
> `disruptor` each time we upgrade it.
>
> Gradle has a special way to treat optional dependencies[3], this
> proposal is the closest Maven-equivalent I could find.
>
> Piotr
>
> [1] https://logging.apache.org/log4j/2.x/manual/async.html#installation
>
> [2] https://logging.apache.org/log4j/2.x/manual/async.html#installation
>
> [3] https://blog.gradle.org/optional-dependencies
>
Re: Create artifacts for optional features
Posted by "Piotr P. Karwasz" <pi...@mailing.copernik.eu>.
Hi Ralph,
On 19.01.2025 17:55, Ralph Goers wrote:
> I must be misunderstanding. IIUC you are proposing that to eliminate optional dependencies we create a bunch of empty jars, presumably with a pom that has the dependency as required. How exactly does that help? Doesn’t that new, empty artifact just become an optional dependency? Isn’t it required during the build to allow the dependency to be resolved?
As you remark, the optional dependencies in `log4j-core` must remain (at
least at compile time, so they must be either `provided` or
`compile+optional` like now). What I am proposing is to create a new
`log4j-async-logger` artifact with a non-optional dependency on
`com.lmax:disruptor`. We should then change the installation
documentation for async loggers[1] and ask users to add:
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-async-logger</artifactId>
<scope>runtime</scope>
</dependency>
The advantage of such a change is that:
* It makes the upgrade to Log4j Core 3 trivial (the same dependency is
required in Log4j Core 3[2])
* Users no longer have to remember what `disruptor` is and why they
added it to their application. They will upgrade the version of
`disruptor` each time we upgrade it.
Gradle has a special way to treat optional dependencies[3], this
proposal is the closest Maven-equivalent I could find.
Piotr
[1] https://logging.apache.org/log4j/2.x/manual/async.html#installation
[2] https://logging.apache.org/log4j/2.x/manual/async.html#installation
[3] https://blog.gradle.org/optional-dependencies
Re: Create artifacts for optional features
Posted by Ralph Goers <ra...@dslextreme.com>.
I must be misunderstanding. IIUC you are proposing that to eliminate optional dependencies we create a bunch of empty jars, presumably with a pom that has the dependency as required. How exactly does that help? Doesn’t that new, empty artifact just become an optional dependency? Isn’t it required during the build to allow the dependency to be resolved?
Ralph
> On Jan 19, 2025, at 2:02 AM, Piotr P. Karwasz <pi...@mailing.copernik.eu> wrote:
>
> Hi all,
>
> Maven is notably bad at handling optional dependencies, which are basically ignored by consumers of the POM file. While Log4j Core 3 solves this problem by not having **any** optional dependencies, we could backport this feature also to Log4j Core 2.
>
> What do you think about adding empty artifacts in Log4j Core 2 for each optional feature? They would work exactly like Spring Boot starters and pull the required modules and dependencies into a user's project. For example `log4j-async-logger` would depend on `com.lmax:disruptor`.
>
> The purpose of such artifacts would be double:
>
> * They would allow a smoother migration to Log4j Core 3 (just a version bump). An empty `log4j-async-logger` from version 2.x, will be replaced with a real JAR in version 3.x.
>
> * They would a semantic purpose: they would allow developers better understand, why a certain dependency is in their stack. This is particularly important in case of vulnerabilities in the dependencies. For example, if the user has `log4j-config-yaml` as dependency and a CVE is published against SnakeYaml, he can easily deduce that the CVE would only affect Log4j Core. Even for an empty `log4j-config-yaml` we can publish a VEX statement that says Log4j Core is not affected, since it does not parse YAML files from untrusted sources. This scenario was discussed on `dev@kafka`[1].
>
> I think we could backport from Log4j Core 3 as empty/starter JARs the following modules:
>
> * `log4j-async-logger` for the Asynchronous Loggers feature.
>
> * `log4j-config-properties` and `log4j-config-yaml`. In the future we might add `log4j-config-xml` in Log4j Core 3 as empty JAR, but with a `requires java.xml` module descriptor. I am not sure if it is worth introducing a `log4j-config-json` in version 2.x, since version 3.x does not require any additional deps for JSON.
>
> * `log4j-compress` for alternative compression algorithms of the rolling file appenders.
>
> * `log4j-conversant` and `log4j-jctools` for alternative queue factories of the Asynchronous Appender.
>
> * `log4j-csv` for the CSV layout.
>
> * `log4j-jndi` for the optional dependency on `java.naming`.
>
> * `log4j-jdbc` and `log4j-jdbc-jndi` for the JDBC Appender and its JNDI connection source.
>
> * `log4j-script` for the optional dependency on `java.scripting`.
>
> Piotr
>
> [1] https://lists.apache.org/thread/bnrr6djsd7nf2135ycpb9xmkwr7c8pt7
>