You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2026/02/24 21:13:37 UTC

[PR] always use StorageMonitor, also make StorageMonitor work for MSQ tasks (druid)

clintropolis opened a new pull request, #19048:
URL: https://github.com/apache/druid/pull/19048

   Makes `StorageMonitor`, added in #18742, be always on. The low overhead seems worth the utility. Also added support for MSQ tasks to work correctly with this monitor, emitting extra `taskId` and `groupId` dimensions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] always use StorageMonitor, also make StorageMonitor work for MSQ tasks (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19048:
URL: https://github.com/apache/druid/pull/19048#discussion_r2880263380


##########
multi-stage-query/src/main/java/org/apache/druid/msq/indexing/IndexerWorkerContext.java:
##########
@@ -152,10 +158,12 @@ public static IndexerWorkerContext createProductionInstance(
   )
   {
     final IndexIO indexIO = injector.getInstance(IndexIO.class);
-    final SegmentManager segmentManager = new SegmentManager(
+    final SegmentCacheManager cacheManager =
         injector.getInstance(SegmentCacheManagerFactory.class)
-                .manufacturate(new File(toolbox.getIndexingTmpDir(), "segment-fetch"), true)
-    );
+                .manufacturate(new File(toolbox.getIndexingTmpDir(), "segment-fetch"), true);
+    final SegmentManager segmentManager = new SegmentManager(cacheManager);
+    final StorageMonitor storageMonitor = new StorageMonitor(cacheManager, Map.of("taskId", task.getId(), "groupId", task.getGroupId()));

Review Comment:
   ah nice, done :+1:



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] always use StorageMonitor, also make StorageMonitor work for MSQ tasks (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #19048:
URL: https://github.com/apache/druid/pull/19048#discussion_r2876680412


##########
multi-stage-query/src/main/java/org/apache/druid/msq/indexing/IndexerWorkerContext.java:
##########
@@ -152,10 +158,12 @@ public static IndexerWorkerContext createProductionInstance(
   )
   {
     final IndexIO indexIO = injector.getInstance(IndexIO.class);
-    final SegmentManager segmentManager = new SegmentManager(
+    final SegmentCacheManager cacheManager =
         injector.getInstance(SegmentCacheManagerFactory.class)
-                .manufacturate(new File(toolbox.getIndexingTmpDir(), "segment-fetch"), true)
-    );
+                .manufacturate(new File(toolbox.getIndexingTmpDir(), "segment-fetch"), true);
+    final SegmentManager segmentManager = new SegmentManager(cacheManager);
+    final StorageMonitor storageMonitor = new StorageMonitor(cacheManager, Map.of("taskId", task.getId(), "groupId", task.getGroupId()));

Review Comment:
   Best to pass in `task::getMetricBuilder` (a supplier of `ServiceMetricEvent.Builder`). It includes `taskId` and `groupId`, also some other stuff like `dataSource`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] always use StorageMonitor, also make StorageMonitor work for MSQ tasks (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm merged PR #19048:
URL: https://github.com/apache/druid/pull/19048


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org