You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "OurNewestMember (via GitHub)" <gi...@apache.org> on 2023/02/08 02:47:04 UTC
[GitHub] [druid] OurNewestMember opened a new issue, #13771: Ingest metrics: messageGap to show earliest and latest messages
OurNewestMember opened a new issue, #13771:
URL: https://github.com/apache/druid/issues/13771
The messageGap metric can be a helpful way to assess the recency of incoming events. For example, an elevated messageGap value could indicate when Druid has little or no lag consuming from a stream, yet the upstream data is delayed. Data recency is also important for observing and planning the required overhead for segment creation during indexing (e.g., "how many segments might this task create?"). This proposal aims to better achieve objectives like these by emitting a metric that helps reflect the dynamic range of the data recency.
Currently the messageGap metric maintains the state of the latest timestamp "seen" by the indexing task
- https://github.com/apache/druid/blob/fb23e38aa716d5f57c4f89352c3e3da5a10ac502/server/src/main/java/org/apache/druid/segment/realtime/FireDepartmentMetrics.java#L275
- https://github.com/apache/druid/blob/fb23e38aa716d5f57c4f89352c3e3da5a10ac502/server/src/main/java/org/apache/druid/segment/realtime/FireDepartmentMetrics.java#L139
This "latest only" behavior conceals the range of incoming event timestamps and therefore does little if anything to represent late arriving data. This greatly limits the utility of the metric for the purposes enumerated above.
This proposal is to emit a "maximum" message gap metric which represents the greatest message gap value occurring over some time. Note that the semantics of the proposal could be somewhat different than for the existing metric because this proposal could require maintaining state not of some event timestamp (current state) but of the computed message gap value.
Outstanding questions/considerations:
- naming: ingest/events/maxMessageGap? (Follows pattern of ingest/kafka/lag versus ingest/kafka/maxLag)
- existing metric name (ingest/events/messageGap): should it be phased out and replaced (eg, with ingest/events/minMessageGap)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Ingest metrics: messageGap to show earliest and latest messages (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13771:
URL: https://github.com/apache/druid/issues/13771#issuecomment-1928547913
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Ingest metrics: messageGap to show earliest and latest messages (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13771:
URL: https://github.com/apache/druid/issues/13771#issuecomment-1979851917
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Ingest metrics: messageGap to show earliest and latest messages (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #13771: Ingest metrics: messageGap to show earliest and latest messages
URL: https://github.com/apache/druid/issues/13771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org