[GitHub] [druid] mghosh4 commented on issue #10352: Metrics reporting system for native parallel batch ingestion

[GitBox <gi...@apache.org>]

mghosh4 commented on issue #10352:
URL: https://github.com/apache/druid/issues/10352#issuecomment-717663587


   Thanks @jihoonson for working on this. This will be super useful. I had a few things that I wanted to draw your attention to:
   1. We have scenarios in our setup where `maxNumConcurrentSubTasks` can become as large as 700 or so. That being said, I do believe having 20 `numConnections` is highly unnecessary considering they will not get any queries (batch ingestion). Do you foresee any issues at this scale?
   2. Another alternate design that I can think of is that the tasks continue to report their metrics to the Overlord (heartbeat) and the supervisor task polls them like it already does to check for health status. It would mean slightly higher memory requirements on Overlord side assuming they might have to store this information. But overlord can probably store aggregates for most metrics. What are other downsides you see? One pro is it does not add any new HTTP connection requirements.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org