You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "abhishekrb19 (via GitHub)" <gi...@apache.org> on 2026/02/06 22:52:09 UTC

[PR] Use LinkedHashMap for EventMap to preserve key-value ordering of Event dimensions (druid)

abhishekrb19 opened a new pull request, #18993:
URL: https://github.com/apache/druid/pull/18993

   Currently, when we use LoggingEmitter and/or LoggingRequestLogger, long, complex SQL queries make up the majority of the log volume, so we truncate logs to some extent. With these events, this can lead to the "value" part, timestamp among other getting truncated since the map's ordering is arbitrary. So this patch switches the EventMap to a LinkedHashMap so critical info (e.g., metric name and value) are logged first, followed by fields like service dimensions, user dimensions and the SQL query itself.
   
   Same idea for the request logger: we log the query context before the SQL query itself, which lets us preserve any useful/debuggable information deterministically even under truncation.
   
   Given these maps typically have at most ~20–30 key/value pairs, the additional overhead of using a LinkedHashMap should be okay.
   
   For example:
   
   ```
   2026-02-06T21:43:41,033 DEBUG [qtp1321214620-64[groupBy_[foo1]_379c91c7-40b1-4d40-b86f-340fcde5177b]] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","timestamp":"2026-02-06T21:43:41.033Z","metric":"query/bytes","value":138,"service":"druid/historical","host":"localhost:8083","version":"37.0.0-SNAPSHOT","context":{"applyLimitPushDown":true,"defaultTimeout":180000,"engine":"native","finalize":false,"fudgeTimestamp":"1454795020784","groupByOutermost":false,"maxQueuedBytes":5242880,"maxScatterGatherBytes":9223372036854775807,"queryFailTime":1770414400967,"queryId":"379c91c7-40b1-4d40-b86f-340fcde5177b","queryResourceId":"c5f4256c-f84f-40c4-964a-840f1505d9a3","resultAsArray":true,"sliceIndex":23,"sqlOuterLimit":1001,"sqlQueryId":"yay","sqlStringifyArrays":false,"timeout":179998},"dataSource":"foo1","duration":"PT2592000S","hasFilters":"false","id":"379c91c7-40b1-4d40-b86f-340fcde5177b","interval":["2025-09-01T00:00:00.000Z/2025-10-01T00:00:00.000Z"
 ],"numComplexMetrics":"0","numDimensions":"3","numMetrics":"0","remoteAddress":"127.0.0.1","statusCode":200,"success":"true","type":"groupBy"}
   ```
   
   instead of 
   
   ```
   org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"server":"localhost:8083","numComplexMetrics":"0","type":"groupBy","version":"37.0.0-SNAPSHOT","duration":"PT2592000S","feed":"metrics","numDimensions":"3","metric":"query/node/ttfb","hasFilters":"false","service":"druid/broker","host":"localhost:8082","context":{"applyLimitPushDown":true,"defaultTimeout":180000,"engine":"native","finalize":false,"fudgeTimestamp":"1675477114564","groupByOutermost":false,"maxQueuedBytes":5242880,"maxScatterGatherBytes":9223372036854775807,"queryFailTime":1770171694634,"queryId":"51152d78-80c6-452b-ba1f-604c2edde74a","queryResourceId":"b5c87e25-8368-4c1c-b974-8fc6e272e54b","resultAsArray":true,"sliceIndex":23,"sqlOuterLimit":1001,"sqlQueryId":"foo","sqlStringifyArrays":false},"interval":["2025-09-01T00:00:00.000Z/2025-10-01T00:00:00.000Z"],"id":"51152d78-80c6-452b-ba1f-604c2edde74a","numMetrics":"0","value":17,"dataSource":"foo1","timestamp":"2026-02-06T02:18:34.654Z"}
   ```
   
   This PR has:
   
   - [x] been self-reviewed.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [x] been tested in a test Druid cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Use LinkedHashMap for EventMap to preserve ordering of Event dimensions (druid)

Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #18993:
URL: https://github.com/apache/druid/pull/18993#discussion_r2783939624


##########
server/src/main/java/org/apache/druid/server/RequestLogLine.java:
##########


Review Comment:
   I don’t think so. The `toString()` and `hashCode()` are typically IDE-generated based on the order of the arguments defined in the constructor. It’s probably only used for debugging and likely predates the `getSqlQueryLine()` and `getNativeQueryLine()` implementations



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Use LinkedHashMap for EventMap to preserve key-value ordering of Event dimensions (druid)

Posted by "aho135 (via GitHub)" <gi...@apache.org>.
aho135 commented on code in PR #18993:
URL: https://github.com/apache/druid/pull/18993#discussion_r2776480490


##########
processing/src/main/java/org/apache/druid/java/util/emitter/core/EventMap.java:
##########
@@ -21,17 +21,18 @@
 
 import com.fasterxml.jackson.databind.annotation.JsonSerialize;
 
-import java.util.HashMap;
+import java.util.LinkedHashMap;
 import java.util.Map;
 
 /**
- * EventMap is a hash map implementation. It can be safely serialzed to JSON using Jackson serializer as it
+ * EventMap is a linked hash map implementation where the insertion order of key-value pairs is maintained.
+ * It can be safely serialzed to JSON using Jackson serializer as it

Review Comment:
   ```suggestion
    * It can be safely serialized to JSON using Jackson serializer as it
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Use LinkedHashMap for EventMap to preserve ordering of Event dimensions (druid)

Posted by "aho135 (via GitHub)" <gi...@apache.org>.
aho135 commented on code in PR #18993:
URL: https://github.com/apache/druid/pull/18993#discussion_r2777871625


##########
processing/src/test/java/org/apache/druid/java/util/emitter/service/ServiceMetricEventTest.java:
##########
@@ -359,4 +362,22 @@ public void test_builder_throwsException_ifDimNameOrValueIsNull()
         () -> eventBuilder.setDimension(null, new String[]{"a"})
     );
   }
+
+  @Test
+  public void testSerializedServiceMetricEventIsOrdered() throws JsonProcessingException
+  {
+    ServiceMetricEvent event = ServiceMetricEvent.builder()
+                                                  .setCreatedTime(DateTimes.of("2026-01-01"))

Review Comment:
   ```suggestion
                                                    .setCreatedTime(DateTimes.of("2026-01-01"))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Use LinkedHashMap for EventMap to preserve key-value ordering of Event dimensions (druid)

Posted by "aho135 (via GitHub)" <gi...@apache.org>.
aho135 commented on code in PR #18993:
URL: https://github.com/apache/druid/pull/18993#discussion_r2776472639


##########
server/src/main/java/org/apache/druid/server/log/DefaultRequestLogEvent.java:
##########
@@ -63,16 +63,15 @@ public EventMap toMap()
         .put("timestamp", getCreatedTime())
         .put("service", getService())
         .put("host", getHost())
+        .put("remoteAddr", getRemoteAddr())
+        .put("queryStats", getQueryStats())
         .putNonNull("query", getQuery());
 
     if (getSql() != null) {
       builder.put("sql", getSql())
              .put("sqlQueryContext", getSqlQueryContext());
     }

Review Comment:
   Can we log the sqlQueryContext before the query itself here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Use LinkedHashMap for EventMap to preserve ordering of Event dimensions (druid)

Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 merged PR #18993:
URL: https://github.com/apache/druid/pull/18993


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Use LinkedHashMap for EventMap to preserve ordering of Event dimensions (druid)

Posted by "aho135 (via GitHub)" <gi...@apache.org>.
aho135 commented on code in PR #18993:
URL: https://github.com/apache/druid/pull/18993#discussion_r2783890338


##########
server/src/main/java/org/apache/druid/server/RequestLogLine.java:
##########


Review Comment:
   Do you think it's worth re-ordering a toString to match the new ordering?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Use LinkedHashMap for EventMap to preserve key-value ordering of Event dimensions (druid)

Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #18993:
URL: https://github.com/apache/druid/pull/18993#discussion_r2776491579


##########
server/src/main/java/org/apache/druid/server/log/DefaultRequestLogEvent.java:
##########
@@ -63,16 +63,15 @@ public EventMap toMap()
         .put("timestamp", getCreatedTime())
         .put("service", getService())
         .put("host", getHost())
+        .put("remoteAddr", getRemoteAddr())
+        .put("queryStats", getQueryStats())
         .putNonNull("query", getQuery());
 
     if (getSql() != null) {
       builder.put("sql", getSql())
              .put("sqlQueryContext", getSqlQueryContext());
     }

Review Comment:
   yes, good point will do that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Use LinkedHashMap for EventMap to preserve ordering of Event dimensions (druid)

Posted by "aho135 (via GitHub)" <gi...@apache.org>.
aho135 commented on code in PR #18993:
URL: https://github.com/apache/druid/pull/18993#discussion_r2783890338


##########
server/src/main/java/org/apache/druid/server/RequestLogLine.java:
##########


Review Comment:
   Do you think it's worth re-ordering toString to match the new ordering?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Use LinkedHashMap for EventMap to preserve ordering of Event dimensions (druid)

Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on PR #18993:
URL: https://github.com/apache/druid/pull/18993#issuecomment-3873378454

   Thanks for the review, @aho135 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org