You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@logging.apache.org by "ramanathan1504 (via GitHub)" <gi...@apache.org> on 2026/07/21 07:59:17 UTC

[PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

ramanathan1504 opened a new pull request, #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186

   Fixes #4166 
   
   Add `keyExcludes` to MDC resolvers to prevent user-controlled variables from overwriting structural log fields.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add literal Context Data (MDC) filtering to JSON Template Layout [logging-log4j2]

Posted by "vy (via GitHub)" <gi...@apache.org>.
vy commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3914140207


##########
log4j-layout-template-json/src/main/resources/EcsLayout.json:
##########
@@ -26,7 +26,19 @@
   "labels": {
     "$resolver": "mdc",
     "flatten": true,
-    "stringified": true
+    "stringified": true,
+    "literal": {
+      "disallowed": [
+        "@timestamp",
+        "message",
+        "log.logger",
+        "log.level",
+        "event.dataset",
+        "process.thread.name",
+        "process.thread.id",
+        "ecs.version"
+      ]
+    }

Review Comment:
   I'd not use the word `filter`, it doesn't match what we already have: `key` and `pattern`, which both "filter".
   
   @ramanathan1504, we had an internal conversation with @ppkarwasz, and [agreed] to overload `key` instead. That is, if `key` is of type string, use the existing behavior, if it is of type object, use the new semantics with `allowed`/`disallowed`. Would you mind updating the PR in that direction, please?
   
   [agreed]: https://the-asf.slack.com/archives/D0394M6GWTX/p1788351977612879?thread_ts=1788344568.252929&cid=D0394M6GWTX



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

Posted by "ramanathan1504 (via GitHub)" <gi...@apache.org>.
ramanathan1504 commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3629069083


##########
log4j-layout-template-json/src/main/java/org/apache/logging/log4j/layout/template/json/resolver/ReadOnlyStringMapResolver.java:
##########
@@ -345,6 +356,9 @@ private enum LoopMethod implements TriConsumer<String, Object, LoopContext> {
 
         @Override
         public void accept(final String key, final Object value, final LoopContext loopContext) {
+            if (loopContext.keyExcludes != null && loopContext.keyExcludes.contains(key)) {

Review Comment:
   @ppkarwasz, 
   fixes 06399c8



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add literal Context Data (MDC) filtering to JSON Template Layout [logging-log4j2]

Posted by "vy (via GitHub)" <gi...@apache.org>.
vy merged PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

Posted by "ramanathan1504 (via GitHub)" <gi...@apache.org>.
ramanathan1504 commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3628504668


##########
log4j-layout-template-json/src/main/java/org/apache/logging/log4j/layout/template/json/resolver/ReadOnlyStringMapResolver.java:
##########
@@ -345,6 +356,9 @@ private enum LoopMethod implements TriConsumer<String, Object, LoopContext> {
 
         @Override
         public void accept(final String key, final Object value, final LoopContext loopContext) {
+            if (loopContext.keyExcludes != null && loopContext.keyExcludes.contains(key)) {

Review Comment:
   are you describing using Collections.emptySet() to avoid the != null check? I can make that change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

Posted by "ppkarwasz (via GitHub)" <gi...@apache.org>.
ppkarwasz commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3628910478


##########
log4j-layout-template-json/src/main/java/org/apache/logging/log4j/layout/template/json/resolver/ReadOnlyStringMapResolver.java:
##########
@@ -345,6 +356,9 @@ private enum LoopMethod implements TriConsumer<String, Object, LoopContext> {
 
         @Override
         public void accept(final String key, final Object value, final LoopContext loopContext) {
+            if (loopContext.keyExcludes != null && loopContext.keyExcludes.contains(key)) {

Review Comment:
   Yes, this should not impact performance and would be easier to understand by developers. You can even add a JSpecify `@NonNull` annotation on the field, so it is clear it can not be `null`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

Posted by "ppkarwasz (via GitHub)" <gi...@apache.org>.
ppkarwasz commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3629969050


##########
log4j-perf-test/src/main/java/org/apache/logging/log4j/perf/jmh/MdcKeyFilterBenchmark.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.logging.log4j.perf.jmh;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+
+@BenchmarkMode(Mode.Throughput)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+@Warmup(iterations = 5, time = 1)
+@Measurement(iterations = 5, time = 1)
+@Fork(1)
+@State(Scope.Benchmark)
+public class MdcKeyFilterBenchmark {

Review Comment:
   We probably should compare the performance of two `ReadOnlyStringMapResolver` instances, instead of comparing `Pattern` vs `Map`. This might give us an idea if the changes are worth it.
   
   `Map` is clearly faster than `Pattern`, but the relative performance gain can not be determined, since it lacks the time needed to format the map as JSON.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

Posted by "ramanathan1504 (via GitHub)" <gi...@apache.org>.
ramanathan1504 commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3630180473


##########
log4j-perf-test/src/main/java/org/apache/logging/log4j/perf/jmh/MdcKeyFilterBenchmark.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.logging.log4j.perf.jmh;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+
+@BenchmarkMode(Mode.Throughput)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+@Warmup(iterations = 5, time = 1)
+@Measurement(iterations = 5, time = 1)
+@Fork(1)
+@State(Scope.Benchmark)
+public class MdcKeyFilterBenchmark {

Review Comment:
   ```text
   Benchmark                                       Mode  Cnt  Score   Error   Units
   MdcKeyFilterBenchmark.testKeyExcludesResolver  thrpt    5  5.049 ± 0.175  ops/us
   MdcKeyFilterBenchmark.testPatternResolver      thrpt    5  1.846 ± 0.023  ops/us
   ```
   @ppkarwasz  5582862



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

Posted by "ramanathan1504 (via GitHub)" <gi...@apache.org>.
ramanathan1504 commented on PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#issuecomment-5440448982

   @vy done — `keyExcludes` is gone, it's `literal` with `allowed` and `disallowed` now, and the grammar block in the `ReadOnlyStringMapResolver` javadoc is updated to match the one u pasted.
   
   Two things your example didn't pin down: `disallowed` currently wins over `allowed`, and when `pattern` is also set a key has to satisfy both — is that what u meant, or should `allowed` widen the pattern instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

Posted by "vy (via GitHub)" <gi...@apache.org>.
vy commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3913049685


##########
src/changelog/.2.x.x/4166_add_mdc_literal_key_filter.xml:
##########
@@ -0,0 +1,13 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<entry xmlns="https://logging.apache.org/xml/ns"
+       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+       xsi:schemaLocation="
+           https://logging.apache.org/xml/ns
+           https://logging.apache.org/xml/ns/log4j-changelog-0.xsd"
+       type="added">
+    <issue id="4166" link="https://github.com/apache/logging-log4j2/issues/4166"/>
+    <issue id="4186" link="https://github.com/apache/logging-log4j2/pull/4186"/>
+    <description format="asciidoc">
+        Add `literal` key filtering to MDC and Map resolvers, and harden `EcsLayout.json` with it.

Review Comment:
   ```suggestion
           Add `literal` key filtering to JSON Template Layout's Context Data (MDC) and Map resolvers.
           Harden `EcsLayout.json` using that.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add literal Context Data (MDC) filtering to JSON Template Layout [logging-log4j2]

Posted by "ppkarwasz (via GitHub)" <gi...@apache.org>.
ppkarwasz commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3913231846


##########
log4j-layout-template-json/src/main/resources/EcsLayout.json:
##########
@@ -26,7 +26,19 @@
   "labels": {
     "$resolver": "mdc",
     "flatten": true,
-    "stringified": true
+    "stringified": true,
+    "literal": {
+      "disallowed": [
+        "@timestamp",
+        "message",
+        "log.logger",
+        "log.level",
+        "event.dataset",
+        "process.thread.name",
+        "process.thread.id",
+        "ecs.version"
+      ]
+    }

Review Comment:
   Code looks good, but the naming for the configuration option seems odd. What about `filter` or `keyFilter`? The word `literal` doesn't tell me much, what this could do.
   
   ```json
   {
     "filter": {
       "includes": [
         "@timestamp",
         ...
       ]
     }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

Posted by "ppkarwasz (via GitHub)" <gi...@apache.org>.
ppkarwasz commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3628381099


##########
log4j-layout-template-json/src/main/java/org/apache/logging/log4j/layout/template/json/resolver/ReadOnlyStringMapResolver.java:
##########
@@ -345,6 +356,9 @@ private enum LoopMethod implements TriConsumer<String, Object, LoopContext> {
 
         @Override
         public void accept(final String key, final Object value, final LoopContext loopContext) {
+            if (loopContext.keyExcludes != null && loopContext.keyExcludes.contains(key)) {

Review Comment:
   What about requiring `loopContext.keyExcludes` to be not null, so we don't need a null-check? The performance should be similar.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] add keyExcludes to MDC resolvers to prevent overwriting of structural… [logging-log4j2]

Posted by "ramanathan1504 (via GitHub)" <gi...@apache.org>.
ramanathan1504 commented on PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#issuecomment-5440452914

   @ppkarwasz reworked to the `literal` grammar @vy proposed on the issue, both your comments are still in — the sets are `Collections.emptySet()` when absent with `@NonNull` on the fields, and the benchmark compares two full layouts: `4.562` vs `1.833 ops/us`.
   
   A mis-shaped `literal` now throws the way `flatten` does, but a typo'd sub-key (`disalowed`) still parses and silently filters nothing — worth rejecting unknown keys under `literal`, or leave it as is?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add literal Context Data (MDC) filtering to JSON Template Layout [logging-log4j2]

Posted by "ramanathan1504 (via GitHub)" <gi...@apache.org>.
ramanathan1504 commented on code in PR #4186:
URL: https://github.com/apache/logging-log4j2/pull/4186#discussion_r3921524575


##########
log4j-layout-template-json/src/main/resources/EcsLayout.json:
##########
@@ -26,7 +26,19 @@
   "labels": {
     "$resolver": "mdc",
     "flatten": true,
-    "stringified": true
+    "stringified": true,
+    "literal": {
+      "disallowed": [
+        "@timestamp",
+        "message",
+        "log.logger",
+        "log.level",
+        "event.dataset",
+        "process.thread.name",
+        "process.thread.id",
+        "ecs.version"
+      ]
+    }

Review Comment:
   @vy [de4101c](https://github.com/apache/logging-log4j2/pull/4186/commits/de4101cddc6a34cace8e095600fe6efe273c764c)  `literal` is gone, `key` takes a string (unchanged) or an object with `allowed`/`disallowed`. The object form combines with `pattern`/`flatten` and `disallowed` wins over `allowed`. tell me if u meant `allowed` to widen the pattern. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org