[GitHub] [druid] kumachanz opened a new issue #11645: Indexer metrics emitter continually emits metrics for completed tasks

[GitBox <gi...@apache.org>]

kumachanz opened a new issue #11645:
URL: https://github.com/apache/druid/issues/11645


   In a cluster using Indexer instead of middle-managers, with metrics emitter enabled, task metrics are emitted forever.
   
   ### Affected Version
   
   0.21.1
   
   ### Description
   
   Please include as much detailed information about the problem as possible.
   
   Cluster runs with combined coordinator and overlord.  There are 10 indexers.
   
   Indexers have been running for 3 days configured with `druid.emitter=logging`, the indexers run a mixture of kafka ingestion and index_parallel ingestion tasks.
   We noticed the amount of log output was increasing in volume as each hour passed.
   
   Looking at the metrics logs we saw this as an example:
   
   ```
   2021-08-30T00:01:27,351 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - 
   {
       "feed": "metrics",
       "timestamp": "2021-08-30T00:01:27.351Z",
       "service": "druid/indexer",
       "host": "10.209.6.173:8091",
       "version": "0.21.1",
       "metric": "ingest/events/thrownAway",
       "value": 0,
       "dataSource": "events",
       "taskId": [
           "partial_index_generate_events_mbbnmdcl_2021-08-27T09:28:56.347Z"
       ]
   }
   ```
   
   
   Notice date of the event is 3 days after the time this partial index was generated.  Searching the logs we found this same metric for the same taskId is emitted every minute since the first time it was generated.
   
   We have worked around this problem by disabling metrics logging on the indexer as there doesn't seem to be a way of having the task metrics emitted once.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org