You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "suneet-s (via GitHub)" <gi...@apache.org> on 2023/08/04 00:43:07 UTC
[PR] Metric to report time spent fetching and analyzing segments (druid)
suneet-s opened a new pull request, #14752:
URL: https://github.com/apache/druid/pull/14752
### Description
Add a new metric that is emitted by the compaction task that shows how much time it spent fetching and analyzing segments. When a datasource is very fragmented, this could be a large amount of time. This metric will be helpful to operators as they can monitor this to see if they would benefit from explicitly specifying the schema in the auto compaction / manual compaction config if their use case allows it.
Here is a sample of what the metric would look like
```
{"taskIngestionMode":"REPLACE_LEGACY","feed":"metrics","taskType":"compact","metric":"compact/segmentAnalyzer/fetchAndProcessMillis","service":"druid/middleManager","groupId":"coordinator-issued_compact_wikipedia_cgmonfmg_2023-08-04T00:35:38.106Z","host":"localhost:8100","version":"28.0.0-SNAPSHOT","value":78,"dataSource":"wikipedia","taskId":"coordinator-issued_compact_wikipedia_cgmonfmg_2023-08-04T00:35:38.106Z","timestamp":"2023-08-04T00:35:40.605Z"}
```
#### Release note
NEW: `compact/segmentAnalyzer/fetchAndProcessMillis` is now reported by compaction tasks to indicate how much time was spent fetching and processing segments to infer the schema
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286201705
##########
docs/operations/metrics.md:
##########
@@ -203,6 +203,14 @@ These metrics apply to the [Kinesis indexing service](../development/extensions-
|`ingest/kinesis/avgLag/time`|Average lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis across all shards. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
|`ingest/kinesis/partitionLag/time`|Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `partition`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
+### Compaction metrics
+
+These metrics are only reported by [compaction tasks](../data-management/compaction.md).
+
+|Metric|Description|Dimensions| Normal value |
+|------|-----------|----------|--------------------------------------------------------------------------------------------------------|
Review Comment:
```suggestion
|Metric|Description|Dimensions|Normal value|
|------|-----------|-----------|------------|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "zachjsh (via GitHub)" <gi...@apache.org>.
zachjsh commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286088451
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -904,6 +928,8 @@ static class ExistingSegmentAnalyzer
// For processMetricsSpec:
private final Set<List<AggregatorFactory>> aggregatorFactoryLists = new HashSet<>();
+ private long processingTimeMillis = 0;
Review Comment:
Is this used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286201705
##########
docs/operations/metrics.md:
##########
@@ -203,6 +203,14 @@ These metrics apply to the [Kinesis indexing service](../development/extensions-
|`ingest/kinesis/avgLag/time`|Average lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis across all shards. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
|`ingest/kinesis/partitionLag/time`|Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `partition`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
+### Compaction metrics
+
+These metrics are only reported by [compaction tasks](../data-management/compaction.md).
+
+|Metric|Description|Dimensions| Normal value |
+|------|-----------|----------|--------------------------------------------------------------------------------------------------------|
Review Comment:
```suggestion
|Metric|Description|Dimensions|Normal value|
|------|-----------|---------|------------|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "suneet-s (via GitHub)" <gi...@apache.org>.
suneet-s merged PR #14752:
URL: https://github.com/apache/druid/pull/14752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286200323
##########
docs/operations/metrics.md:
##########
@@ -203,6 +203,14 @@ These metrics apply to the [Kinesis indexing service](../development/extensions-
|`ingest/kinesis/avgLag/time`|Average lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis across all shards. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
|`ingest/kinesis/partitionLag/time`|Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `partition`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
+### Compaction metrics
+
+These metrics are only reported by [compaction tasks](../data-management/compaction.md).
+
+|Metric|Description|Dimensions| Normal value |
Review Comment:
```suggestion
|Metric|Description|Dimensions|Normal value|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "suneet-s (via GitHub)" <gi...@apache.org>.
suneet-s commented on PR #14752:
URL: https://github.com/apache/druid/pull/14752#issuecomment-1668698952
Thanks @ektravel and @zachjsh for the reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286199533
##########
docs/operations/metrics.md:
##########
@@ -203,6 +203,14 @@ These metrics apply to the [Kinesis indexing service](../development/extensions-
|`ingest/kinesis/avgLag/time`|Average lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis across all shards. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
|`ingest/kinesis/partitionLag/time`|Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `partition`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
+### Compaction metrics
+
+These metrics are only reported by [compaction tasks](../data-management/compaction.md).
Review Comment:
```suggestion
[Compaction tasks](../data-management/compaction.md) emit the following metrics.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286204064
##########
docs/operations/metrics.md:
##########
@@ -203,6 +203,14 @@ These metrics apply to the [Kinesis indexing service](../development/extensions-
|`ingest/kinesis/avgLag/time`|Average lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis across all shards. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
|`ingest/kinesis/partitionLag/time`|Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `partition`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
+### Compaction metrics
+
+These metrics are only reported by [compaction tasks](../data-management/compaction.md).
+
+|Metric|Description|Dimensions| Normal value |
+|------|-----------|----------|--------------------------------------------------------------------------------------------------------|
+|`compact/segmentAnalyzer/fetchAndProcessMillis`|Time taken to fetch and process segments to infer the schema for the compaction task to run.|`dataSource`, `taskId`, `taskType`, `groupId`,`tags`| Varies. A high value indicates compaction tasks will speed up from explicitly setting the data schema. |
Review Comment:
```suggestion
|`compact/segmentAnalyzer/fetchAndProcessMillis`|The time it takes to fetch and process segments to infer the schema for the compaction task to run.|`dataSource`, `taskId`, `taskType`, `groupId`,`tags`| Varies. A high value indicates compaction tasks will speed up from explicitly setting the data schema. |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286201705
##########
docs/operations/metrics.md:
##########
@@ -203,6 +203,14 @@ These metrics apply to the [Kinesis indexing service](../development/extensions-
|`ingest/kinesis/avgLag/time`|Average lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis across all shards. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
|`ingest/kinesis/partitionLag/time`|Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `partition`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
+### Compaction metrics
+
+These metrics are only reported by [compaction tasks](../data-management/compaction.md).
+
+|Metric|Description|Dimensions| Normal value |
+|------|-----------|----------|--------------------------------------------------------------------------------------------------------|
Review Comment:
```suggestion
|------|-----------|----------|------------|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "suneet-s (via GitHub)" <gi...@apache.org>.
suneet-s commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286096469
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -904,6 +928,8 @@ static class ExistingSegmentAnalyzer
// For processMetricsSpec:
private final Set<List<AggregatorFactory>> aggregatorFactoryLists = new HashSet<>();
+ private long processingTimeMillis = 0;
Review Comment:
removed. I had thought perhaps a metric for all the time spent in this class would be interesting, but decided against implementing it as it would have resulted in a few new metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286204064
##########
docs/operations/metrics.md:
##########
@@ -203,6 +203,14 @@ These metrics apply to the [Kinesis indexing service](../development/extensions-
|`ingest/kinesis/avgLag/time`|Average lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis across all shards. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
|`ingest/kinesis/partitionLag/time`|Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `partition`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
+### Compaction metrics
+
+These metrics are only reported by [compaction tasks](../data-management/compaction.md).
+
+|Metric|Description|Dimensions| Normal value |
+|------|-----------|----------|--------------------------------------------------------------------------------------------------------|
+|`compact/segmentAnalyzer/fetchAndProcessMillis`|Time taken to fetch and process segments to infer the schema for the compaction task to run.|`dataSource`, `taskId`, `taskType`, `groupId`,`tags`| Varies. A high value indicates compaction tasks will speed up from explicitly setting the data schema. |
Review Comment:
```suggestion
|`compact/segmentAnalyzer/fetchAndProcessMillis`|The time it takes to fetch and process segments to infer the schema for the compaction task to run.|`dataSource`, `taskId`, `taskType`, `groupId`,`tags`| Varies. A high value indicates compaction tasks will speed up from explicitly setting the data schema. |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Metric to report time spent fetching and analyzing segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14752:
URL: https://github.com/apache/druid/pull/14752#discussion_r1286201705
##########
docs/operations/metrics.md:
##########
@@ -203,6 +203,14 @@ These metrics apply to the [Kinesis indexing service](../development/extensions-
|`ingest/kinesis/avgLag/time`|Average lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis across all shards. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
|`ingest/kinesis/partitionLag/time`|Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.|`dataSource`, `stream`, `partition`, `tags`|Greater than 0, up to max Kinesis retention period in milliseconds. |
+### Compaction metrics
+
+These metrics are only reported by [compaction tasks](../data-management/compaction.md).
+
+|Metric|Description|Dimensions| Normal value |
+|------|-----------|----------|--------------------------------------------------------------------------------------------------------|
Review Comment:
```suggestion
|Metric|Description|Dimensions|Normal value|
|------|-----------|------------|-------------|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org