You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rhodo (via GitHub)" <gi...@apache.org> on 2025/03/21 19:59:58 UTC

[PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/'s getChildren response size [pinot]
rhodo opened a new pull request, #15345:
URL: https://github.com/apache/pinot/pull/15345

   Currently, we lack visibility into how much large the getChildren() response could be under "/PROPERTYSTORE/SEGMENTS/\<table\>" especially when the number of segments is huge.
   
   This PR adds a new metric PROPERTYSTORE_SEGMENT_CHILDREN_BYTE_SIZE to track the total bytes used by segment names for each table in ZooKeeper's PropertyStore.
   
   Modified existing integration tests in ControllerPeriodicTasksIntegrationTest to verify the new metric is correctly calculated and reported.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/

's getChildren response size [pinot]Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #15345:
URL: https://github.com/apache/pinot/pull/15345#issuecomment-2752340040

   Can you fix the tests in `SegmentStatusCheckerTest`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/

's getChildren response size [pinot]Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15345:
URL: https://github.com/apache/pinot/pull/15345#issuecomment-2744499959

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15345?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 55.94%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`572d1a3`)](https://app.codecov.io/gh/apache/pinot/commit/572d1a3e8327d8817d404398ec40015e96854b7e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 1891 commits behind head on master.
   
   > :exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (572d1a3). Click for more details.
   > 
   > <details><summary>HEAD has 53 uploads less than BASE</summary>
   >
   >| Flag | BASE (59551e4) | HEAD (572d1a3) |
   >|------|------|------|
   >|integration|7|0|
   >|integration2|3|0|
   >|temurin|12|1|
   >|java-21|7|1|
   >|skip-bytebuffers-true|3|0|
   >|skip-bytebuffers-false|7|1|
   >|unittests|5|1|
   >|unittests1|2|1|
   >|java-11|5|0|
   >|unittests2|3|0|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #15345      +/-   ##
   ============================================
   - Coverage     61.75%   55.94%   -5.81%     
   - Complexity      207      685     +478     
   ============================================
     Files          2436     2174     -262     
     Lines        133233   117276   -15957     
     Branches      20636    18750    -1886     
   ============================================
   - Hits          82274    65611   -16663     
   - Misses        44911    46255    +1344     
   + Partials       6048     5410     -638     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.94% <100.00%> (-5.68%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.94% <100.00%> (-5.81%)` | :arrow_down: |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.94% <100.00%> (-5.81%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.94% <100.00%> (-5.81%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.94% <100.00%> (+9.05%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15345?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/

's getChildren response size [pinot]Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #15345:
URL: https://github.com/apache/pinot/pull/15345


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/

's getChildren response size [pinot]Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on PR #15345:
URL: https://github.com/apache/pinot/pull/15345#issuecomment-2750282531

   > > One concern is that constantly reading the segment list of a large table itself may put pressure on the controller and zookeeper. Is it possible to track this metric when segments are added or removed ? A daily periodic job can refresh the absolute value if required. Another suggestion is to add a config param as getting the list is a heavy operation.
   > 
   > From reading the code, it looks like ZooKeeper access via the property store’s getChildrenNames() method is already cached ([ref](https://sourcegraph.com/github.com/apache/helix/-/blob/helix-core/src/main/java/org/apache/helix/manager/zk/ZkCacheBaseDataAccessor.java?L682)). Also, segment status checkers run every 5 minutes by default. Let me know if this addresses your concern.
   
   Got it. Didnt realize this is iterating through an in-memory cached list


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/

's getChildren response size [pinot]Posted by "rhodo (via GitHub)" <gi...@apache.org>.
rhodo commented on PR #15345:
URL: https://github.com/apache/pinot/pull/15345#issuecomment-2749267004

   > One concern is that constantly reading the segment list of a large table itself may put pressure on the controller and zookeeper. Is it possible to track this metric when segments are added or removed ? A daily periodic job can refresh the absolute value if required. Another suggestion is to add a config param as getting the list is a heavy operation.
   
   From reading the code, it looks like ZooKeeper access via the property store’s getChildrenNames() method is already cached ([ref](https://sourcegraph.com/github.com/apache/helix/-/blob/helix-core/src/main/java/org/apache/helix/manager/zk/ZkCacheBaseDataAccessor.java?L682)). Also, segment status checkers run every 5 minutes by default. Let me know if this addresses your concern.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/

's getChildren response size [pinot]Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on PR #15345:
URL: https://github.com/apache/pinot/pull/15345#issuecomment-2747816587

   One concern is that constantly reading the segment list of a large table itself may put pressure on the controller and zookeeper. Is it possible to track this metric when segments are added or removed ? A daily periodic job can refresh the absolute value if required.
   Another suggestion is to add a config param as getting the list is a heavy operation. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/

's getChildren response size [pinot]Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on PR #15345:
URL: https://github.com/apache/pinot/pull/15345#issuecomment-2756672665

   @rhodo 
   I understand this PR has already been merged but would like to understand following
   1. What is the rationale of adding this metric. Where would we need this ?
   2.  The constant name PROPERTYSTORE_SEGMENT_CHILDREN_BYTE_SIZE gives a perception that it's the size of the child node and not just the segment name sizes. Can we update the name to maybe `PROPERTYSTORE_SEGMENT_CHILDREN_NAME_BYTE_SIZE` ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/

's getChildren response size [pinot]Posted by "rhodo (via GitHub)" <gi...@apache.org>.
rhodo commented on PR #15345:
URL: https://github.com/apache/pinot/pull/15345#issuecomment-2752666256

   > Can you fix the tests in `SegmentStatusCheckerTest`?
   
   Sure, just fixed. It is failing because propertyStore can be null in SegmentStatusCheckerTest.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/

's getChildren response size [pinot]Posted by "rhodo (via GitHub)" <gi...@apache.org>.
rhodo commented on PR #15345:
URL: https://github.com/apache/pinot/pull/15345#issuecomment-2759774679

   > @rhodo I understand this PR has already been merged but would like to understand following
   > 
   > 1. What is the rationale of adding this metric. Where would we need this ?
   > 2. The constant name PROPERTYSTORE_SEGMENT_CHILDREN_BYTE_SIZE gives a perception that it's the size of the child node and not just the segment name sizes. Can we update the name to maybe `PROPERTYSTORE_SEGMENT_CHILDREN_NAME_BYTE_SIZE` ?
   
   The rationale is that when the number of segments becomes very large, the getChildren() call could fail or impact availability if the returned list exceeds ZooKeeper's buffer size. This metric helps us monitor and stay ahead of that risk.
   
   I don’t have a strong opinion on the naming. I was following the existing convention (e.g., IDEALSTATE_ZNODE_BYTE_SIZE for node size metrics), so I thought PROPERTYSTORE_SEGMENT_ZNODE_BYTE_SIZE might be appropriate if we're tracking znode size. But if you feel strongly about renaming it to something like PROPERTYSTORE_SEGMENT_CHILDREN_NAME_BYTE_SIZE, I’m happy to make that change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org