You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "ElliotXue-crypto (via GitHub)" <gi...@apache.org> on 2026/08/25 09:12:05 UTC
[I] [Bug] [skywalking]
ElliotXue-crypto opened a new issue, #14006:
URL: https://github.com/apache/skywalking/issues/14006
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.
### Apache SkyWalking Component
Java Agent (apache/skywalking-java)
### What happened
Since **9.5.0**, the released `apm-toolkit-log4j-2.x` jar no longer contains the Log4j2 plugin descriptor:
```
META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat
```
Without that descriptor Log4j2 cannot discover `TraceIdConverter`, so `%traceId` in a `PatternLayout` is greedily matched as the built-in `%t` (thread name) followed by the literal `raceId`:
```
ERROR [main] [mainraceId] Foo: message <- actual
ERROR [main] [TID: N/A] Foo: message <- expected
```
Verified directly against Maven Central:
| Version | `Log4j2Plugins.dat` |
|---------|---------------------|
| 9.0.0 – 9.4.0 | present |
| 9.5.0 – 9.7.0 | **missing** |
Only `apm-toolkit-log4j-2.x` is affected. `apm-toolkit-log4j-1.x` and `apm-toolkit-logback-1.x` never shipped this file, as they do not use the Log4j2 plugin mechanism.
### Root cause
The toolkit module itself did not change:
- `apm-toolkit-log4j-2.x/pom.xml` is identical between 9.4.0 and 9.5.0 apart from the parent `<version>`.
- Every converter/appender class is byte-identical (sha1, first 12 chars):
| class | 9.4.0 | 9.6.0 |
|-------|-------|-------|
| `TraceIdConverter` | `76bf70b1d17c` | `76bf70b1d17c` |
| `SkyWalkingContextConverter` | `c5481fdab754` | `c5481fdab754` |
| `Log4j2OutputAppender` | `79bc1667fbac` | `79bc1667fbac` |
| `Log4j2SkyWalkingContextOutputAppender` | `cbe0b6076cee` | `cbe0b6076cee` |
The change is in the **`java-agent` root pom**. 9.5.0 added an `annotationProcessorPaths` block to `maven-compiler-plugin`:
```xml
<annotationProcessorPaths>
<path>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
</path>
</annotationProcessorPaths>
```
Once `annotationProcessorPaths` is declared explicitly, javac stops discovering annotation processors from the compile classpath. `apm-toolkit-log4j-2.x` declares `log4j-core` as `provided`, and it is Log4j2's `PluginProcessor` — previously picked up from that classpath — that generates `Log4j2Plugins.dat`. With the processor list narrowed to Lombok only, `PluginProcessor` never runs and the descriptor is silently dropped.
This matches the 9.5.0 changelog entry:
> Centralized plugin version management in the root POM and remove redundant declarations
so this looks like an unintended side effect of a build refactor rather than a deliberate removal.
### What you expected to happen
The released jar should contain `Log4j2Plugins.dat`, so that `%traceId` and `%sw_ctx` resolve automatically as documented, without users having to declare `packages` in `log4j2.xml`.
### How to reproduce
Confirm the descriptor is missing:
```bash
for v in 9.4.0 9.5.0 9.6.0 9.7.0; do
curl -sfLO "https://repo1.maven.org/maven2/org/apache/skywalking/apm-toolkit-log4j-2.x/$v/apm-toolkit-log4j-2.x-$v.jar"
printf "%-8s dat=%s\n" "$v" "$(unzip -l apm-toolkit-log4j-2.x-$v.jar | grep -c Log4j2Plugins.dat)"
done
```
Output:
```
9.4.0 dat=1
9.5.0 dat=0
9.6.0 dat=0
9.7.0 dat=0
```
End-to-end, with `apm-toolkit-log4j-2.x:9.6.0` and `log4j-core:2.24.3` on the classpath and this `log4j2.xml`:
```xml
<Configuration status="warn">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%p [%t] [%traceId] %c{1.}: %m%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="DEBUG"><AppenderRef ref="Console"/></Root>
</Loggers>
</Configuration>
```
a single `log.error("MARKER")` prints `ERROR [main] [mainraceId] ...`. Swapping the toolkit to 9.4.0, with no other change, prints `ERROR [main] [TID: N/A] ...`.
### Anything else
**Impact.** Every project on 9.5.0+ that uses `%traceId` or `%sw_ctx` in `log4j2.xml`. The failure is silent — logging keeps working and only the trace id is replaced by a thread name, so it is easy to ship without noticing. This has been broken across three releases (9.5.0, 9.6.0, 9.7.0).
**Workarounds**, both verified locally:
1. Declare the package explicitly. Note `packages` is deprecated in Log4j2 and removed in Log4j 3:
```xml
<Configuration packages="org.apache.skywalking.apm.toolkit.log.log4j.v2.x">
```
2. Pin the toolkit back to 9.4.0. Since all classes are byte-identical, this is functionally equivalent to 9.6.0 plus the missing descriptor.
**Suggested fix.** Add `log4j-core` to the processor paths for this module. Because `annotationProcessorPaths` replaces rather than appends when overriding the parent, the Lombok entry has to be repeated:
```xml
<!-- apm-toolkit-log4j-2.x/pom.xml -->
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<annotationProcessorPaths>
<path>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
</path>
<path>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j-core.version}</version>
</path>
</annotationProcessorPaths>
</configuration>
</plugin>
</plugins>
</build>
```
It would also be worth adding a build-time check that the descriptor exists in the packaged jar, so a future refactor cannot drop it silently again.
**Disclosure.** The root-cause analysis above was done with the help of an AI assistant (Claude). Every claim in it — the version boundary, the class checksums, the pom diff, and both workarounds — was verified against artifacts published on Maven Central and can be reproduced with the commands above.
### Are you willing to submit a pull request to fix on your own?
- [ ] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] apm-toolkit-log4j-2.x jar has been missing Log4j2Plugins.dat since 9.5.0, breaking %traceId, %sw_ctx and GRPCLogClientAppender [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #14006: [Bug] apm-toolkit-log4j-2.x jar has been missing Log4j2Plugins.dat since 9.5.0, breaking %traceId, %sw_ctx and GRPCLogClientAppender
URL: https://github.com/apache/skywalking/issues/14006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] apm-toolkit-log4j-2.x jar has been missing Log4j2Plugins.dat since 9.5.0, breaking %traceId, %sw_ctx and GRPCLogClientAppender [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #14006:
URL: https://github.com/apache/skywalking/issues/14006#issuecomment-5408783369
Thanks for finding this and reporting this. This is an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org