You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "lnbest0707-uber (via GitHub)" <gi...@apache.org> on 2024/01/29 18:41:42 UTC
[PR] Add metrics for no-HA segments monitoring [pinot]
lnbest0707-uber opened a new pull request, #12336:
URL: https://github.com/apache/pinot/pull/12336
Summary:
Add metrics to monitor any segments running with only one replica. This could help monitor the reliability risk, especially during real world node replacement.
Reasons for not using existing metrics:
- PERCENT_SEGMENTS_AVAILABLE is tracking difference between nOffline and nSegments. Where nOffline refers to nReplicas == 0. In node replacement workflow, we need to track the number of segments whose nReplica==1 which does not have any representation now.
- PERCENT_OF_REPLICAS only tells if any segment in the table is unavailable. It is counting segment with lowest percentage of replica running.
For example, if a table has 2000 segments, 1999 of them are with 2 replicas, 1 of them is with 1 replica. The metrics will show 50%.
In another example, if 1000 of them are with 2 replicas, the other 1000 of them are with 1 replica. The metrics will still show 50%.
But apparently those 2 cases have significantly different level of risks. We need differ them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for no-HA segments monitoring [pinot]
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #12336:
URL: https://github.com/apache/pinot/pull/12336#discussion_r1473582032
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java:
##########
@@ -311,6 +312,9 @@ private void updateSegmentMetrics(String tableNameWithType, TableConfig tableCon
LOGGER.warn("Segment {} of table {} has no online replicas", partitionName, tableNameWithType);
}
nOffline++;
+ } else if (nReplicas == 1) {
Review Comment:
should we check if the segment is configured to be single replica in table config?
If so, i dont think we should emit any metrics (e.g. only emit this when the nReplicas is smaller than the expected one)
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java:
##########
@@ -323,6 +327,8 @@ private void updateSegmentMetrics(String tableNameWithType, TableConfig tableCon
_controllerMetrics.setValueOfTableGauge(tableNameWithType, ControllerGauge.PERCENT_OF_REPLICAS,
(nReplicasIdealMax > 0) ? (nReplicasExternal * 100 / nReplicasIdealMax) : 100);
_controllerMetrics.setValueOfTableGauge(tableNameWithType, ControllerGauge.SEGMENTS_IN_ERROR_STATE, nErrors);
+ _controllerMetrics.setValueOfTableGauge(tableNameWithType, ControllerGauge.SEGMENTS_WITHOUT_HIGH_AVAILABILITY,
+ nNoHA);
Review Comment:
i was wondering if we should name: SEGMENTS_WITH_SINGLE_REPLICA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for no-HA segments monitoring [pinot]
Posted by "chenboat (via GitHub)" <gi...@apache.org>.
chenboat commented on PR #12336:
URL: https://github.com/apache/pinot/pull/12336#issuecomment-1922689613
> > the approach looks good. i was wondering if we should have some definitions. e.g. when nReplica equals the following, how should we call each status. HighAvailability is not a very well defined term
> > ```
> > - 0 : OFFLINE
> > - 1 : ??
> > - expected - 1 : ??
> > - expected : HEALTHY
> > ```
>
> Thanks for the review. Yes, having a well defined definition is important. The "high availability" term I am using was refer to [hadoop HA definition](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html), which means running with 2 or more nodes for the same instance as active/standby(s). The issue is not exactly same as here and I am open to the naming.
>
> Theoretically, running with expected - 1 is what we always observe during real world node replacement. Probably track "actual <= expected - 1" as SEGMENTS_WITH_LESS_REPLICAS? And running with only 1 replica has far more risks than other <expected values (e.g. 2, 3 ...). Not sure if it is a overkill to track it separately.
I think SEGMENTS_WITH_REDUCED_CAPACITY OR LESS_RESPLICAS is what is useful for admin. This metrics should alert users when there is a prolonged time of reduced capacity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for no-HA segments monitoring [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on PR #12336:
URL: https://github.com/apache/pinot/pull/12336#issuecomment-1920154349
> the approach looks good. i was wondering if we should have some definitions. e.g. when nReplica equals the following, how should we call each status. HighAvailability is not a very well defined term
>
> ```
> - 0 : OFFLINE
> - 1 : ??
> - expected - 1 : ??
> - expected : HEALTHY
> ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for no-HA segments monitoring [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on PR #12336:
URL: https://github.com/apache/pinot/pull/12336#issuecomment-1920166282
> the approach looks good. i was wondering if we should have some definitions. e.g. when nReplica equals the following, how should we call each status. HighAvailability is not a very well defined term
>
> ```
> - 0 : OFFLINE
> - 1 : ??
> - expected - 1 : ??
> - expected : HEALTHY
> ```
Thanks for the review.
Yes, having a well defined definition is important. The "high availability" term I am using was refer to [hadoop HA definition](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html), which means running with 2 or more nodes for the same instance as active/standby(s). The issue is not exactly same as here and I am open to the naming.
Theoretically, running with expected - 1 is what we always observe during real world node replacement. Probably track "actual <= expected - 1" as SEGMENTS_WITH_LESS_REPLICAS?
And running with only 1 replica has far more risks than other <expected values (e.g. 2, 3 ...). Not sure if it is a overkill to track it separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for no-HA segments monitoring [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12336:
URL: https://github.com/apache/pinot/pull/12336#issuecomment-1919783914
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12336?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `10 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`2acf8ea`)](https://app.codecov.io/gh/apache/pinot/commit/2acf8ea354b30e2e8415cb763bae34f37cf26c6a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.61% compared to head [(`ac23642`)](https://app.codecov.io/gh/apache/pinot/pull/12336?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 34.74%.
> Report is 8 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/12336?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...e/pinot/controller/helix/SegmentStatusChecker.java](https://app.codecov.io/gh/apache/pinot/pull/12336?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9TZWdtZW50U3RhdHVzQ2hlY2tlci5qYXZh) | 0.00% | [9 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12336?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/pinot/common/metrics/ControllerGauge.java](https://app.codecov.io/gh/apache/pinot/pull/12336?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyR2F1Z2UuamF2YQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12336?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12336 +/- ##
=============================================
- Coverage 61.61% 34.74% -26.88%
+ Complexity 1152 944 -208
=============================================
Files 2421 2345 -76
Lines 131839 128134 -3705
Branches 20344 19790 -554
=============================================
- Hits 81239 44526 -36713
- Misses 44647 80416 +35769
+ Partials 5953 3192 -2761
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.74% <0.00%> (-26.76%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.74% <0.00%> (-26.74%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.74% <0.00%> (-26.88%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.56% <0.00%> (-15.06%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.56% <0.00%> (-0.21%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12336?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for no-HA segments monitoring [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on PR #12336:
URL: https://github.com/apache/pinot/pull/12336#issuecomment-1924507664
> > > the approach looks good. i was wondering if we should have some definitions. e.g. when nReplica equals the following, how should we call each status. HighAvailability is not a very well defined term
> > > ```
> > > - 0 : OFFLINE
> > > - 1 : ??
> > > - expected - 1 : ??
> > > - expected : HEALTHY
> > > ```
> >
> >
> > Thanks for the review. Yes, having a well defined definition is important. The "high availability" term I am using was refer to [hadoop HA definition](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html), which means running with 2 or more nodes for the same instance as active/standby(s). The issue is not exactly same as here and I am open to the naming.
> > Theoretically, running with expected - 1 is what we always observe during real world node replacement. Probably track "actual <= expected - 1" as SEGMENTS_WITH_LESS_REPLICAS? And running with only 1 replica has far more risks than other <expected values (e.g. 2, 3 ...). Not sure if it is a overkill to track it separately.
>
> I think SEGMENTS_WITH_REDUCED_CAPACITY OR LESS_RESPLICAS is what is useful for admin. This metrics should alert users when there is a prolonged time of reduced capacity.
Yes, switched to "SEGMENTS_WITH_LESS_REPLICAS".
The failure in the integration test looks not related..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for SEGMENTS_WITH_LESS_REPLICAS monitoring [pinot]
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on code in PR #12336:
URL: https://github.com/apache/pinot/pull/12336#discussion_r1486707499
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java:
##########
@@ -323,9 +330,12 @@ private void updateSegmentMetrics(String tableNameWithType, TableConfig tableCon
if (nOffline > 0) {
LOGGER.warn("Table {} has {} segments with no online replicas", tableNameWithType, nOffline);
}
+ if (nLessReplicas > 0) {
+ LOGGER.info("Table {} has {} segments with no high availability", tableNameWithType, nLessReplicas);
+ }
if (nReplicasExternal < nReplicasIdealMax) {
- LOGGER.warn("Table {} has {} replicas, below replication threshold :{}", tableNameWithType, nReplicasExternal,
- nReplicasIdealMax);
+ LOGGER.warn("Table {} has {} replicas, below replication threshold :{}, with {} segments running with less than"
Review Comment:
What do we mean by "Table foo has x replicas"? Can we say: "Table foo has at least one segment with x replicas...."
Also, you don't need to log `nLessReplicas` here since you are logging it above anyways?
I think we should also rename `nReplicasExternal` to `minReplicasOfAnySegment` to make this more readable.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java:
##########
@@ -323,9 +330,12 @@ private void updateSegmentMetrics(String tableNameWithType, TableConfig tableCon
if (nOffline > 0) {
LOGGER.warn("Table {} has {} segments with no online replicas", tableNameWithType, nOffline);
}
+ if (nLessReplicas > 0) {
+ LOGGER.info("Table {} has {} segments with no high availability", tableNameWithType, nLessReplicas);
Review Comment:
Let's reword this to be explicit. "with number of replicas less than the replication factor"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for SEGMENTS_WITH_LESS_REPLICAS monitoring [pinot]
Posted by "chenboat (via GitHub)" <gi...@apache.org>.
chenboat merged PR #12336:
URL: https://github.com/apache/pinot/pull/12336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for SEGMENTS_WITH_LESS_REPLICAS monitoring [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on PR #12336:
URL: https://github.com/apache/pinot/pull/12336#issuecomment-1928495920
> sounds good. so do we have
>
> * 0 : OFFLINE
> * 1 : ??
> * expected - 1 : SEGMENTS_WITH_LESS_REPLICAS
> * expected : HEALTHY
>
> am I correct?
Yes, correct, right now the second (1) case is not treated separately.
> 1. how do we plan to avoid these alerts during say rebalance or server upgrade? (it will be one less during operation which might be a noise to monitoring system)
The patch only produces metrics, and alert wise relies on how the admin monitoring the system. From my point of view, it could be based on the number and duration of SEGMENTS_WITH_LESS_REPLICAS.
For example, to improve the node replacement operation. The key metrics is **how many** segments are running with SEGMENTS_WITH_LESS_REPLICAS for **how long**. And then in the real operation, if this metrics runs over the threshold we set (by experiment), it could indicate something unusual happened.
Same for rebalancing and restart, if it goes over some threshold, then either something goes wrong or the existing workflow needs to be improved.
> 2. do we want to address the naming of the "only 1 replica available, where configured to have more than 1" ?
Right now, I do not have enough data to justify if it is critical to tell "only 1 replica available". In most of cases I saw, monitoring SEGMENTS_WITH_LESS_REPLICAS could uncover most of issues.
> 3. what if the RF is set to two? do we honor "only 1 replica" or "1 less replica" or both?
If replica is set to 2, then 1 == expected - 1, the SEGMENTS_WITH_LESS_REPLICAS in the patch could tell both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for SEGMENTS_WITH_LESS_REPLICAS monitoring [pinot]
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on PR #12336:
URL: https://github.com/apache/pinot/pull/12336#issuecomment-1928211216
sounds good. so do we have
- 0 : OFFLINE
- 1 : ??
- expected - 1 : SEGMENTS_WITH_LESS_REPLICAS
- expected : HEALTHY
am I correct?
several questions
1. how do we plan to avoid these alerts during say rebalance or server upgrade? (it will be one less during operation which might be a noise to monitoring system)
2. do we want to address the naming of the "only 1 replica available, where configured to have more than 1" ?
3. what if the RF is set to two? do we honor "only 1 replica" or "1 less replica" or both?
(btw question 2/3 can be answered totally separate from this PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for SEGMENTS_WITH_LESS_REPLICAS monitoring [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on code in PR #12336:
URL: https://github.com/apache/pinot/pull/12336#discussion_r1486777455
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java:
##########
@@ -323,9 +330,12 @@ private void updateSegmentMetrics(String tableNameWithType, TableConfig tableCon
if (nOffline > 0) {
LOGGER.warn("Table {} has {} segments with no online replicas", tableNameWithType, nOffline);
}
+ if (nLessReplicas > 0) {
+ LOGGER.info("Table {} has {} segments with no high availability", tableNameWithType, nLessReplicas);
+ }
if (nReplicasExternal < nReplicasIdealMax) {
- LOGGER.warn("Table {} has {} replicas, below replication threshold :{}", tableNameWithType, nReplicasExternal,
- nReplicasIdealMax);
+ LOGGER.warn("Table {} has {} replicas, below replication threshold :{}, with {} segments running with less than"
Review Comment:
It was logged as this because the above on was info not warn.
Changed that one to warn and remove logging nLessReplicas here.
> What do we mean by "Table foo has x replicas"? Can we say: "Table foo has at least one segment with x replicas...."
Yes, correct. Made the changed.
> I think we should also rename nReplicasExternal to minReplicasOfAnySegment to make this more readable.
Agree. But did not add that in this patch as it has more reference occurrence to be changed. Will probably revise in another patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for no-HA segments monitoring [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber closed pull request #12336: Add metrics for no-HA segments monitoring
URL: https://github.com/apache/pinot/pull/12336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add metrics for SEGMENTS_WITH_LESS_REPLICAS monitoring [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on code in PR #12336:
URL: https://github.com/apache/pinot/pull/12336#discussion_r1486777720
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java:
##########
@@ -323,9 +330,12 @@ private void updateSegmentMetrics(String tableNameWithType, TableConfig tableCon
if (nOffline > 0) {
LOGGER.warn("Table {} has {} segments with no online replicas", tableNameWithType, nOffline);
}
+ if (nLessReplicas > 0) {
+ LOGGER.info("Table {} has {} segments with no high availability", tableNameWithType, nLessReplicas);
Review Comment:
Good catch, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org