You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "sylph-eu (via GitHub)" <gi...@apache.org> on 2023/09/11 11:45:48 UTC

[GitHub] [pinot] sylph-eu opened a new pull request, #11565: [bugfix] Reduce noise level due to slow topics

sylph-eu opened a new pull request, #11565:
URL: https://github.com/apache/pinot/pull/11565

   Our deployment of Pinot has plenty of "slow" REALTIME table for which no new data are produced for several minutes in a row. As the result, stream consumers are regularly recreated with the following WARN message
   
   ```
   Recreating stream consumer for topic partition <kafka-topic>-<partition>, reason: Total idle time: 183623 ms exceeded idle timeout: 180000 ms
   ```
   
   I'd like to reduce the level of this message to drop plenty of noisy WARN+ messages that pollutes our logging system.
   
   Speaking of the actual usage, this message inside `recreateStreamConsumer` being called from two places:
   1. From `handleTransientStreamErrors`, when max consecutive error count exceeds the threshold. In this case there's another warning message `_segmentLogger
             .warn("Stream transient exception when fetching messages, retrying (count={})", _consecutiveErrorCount, e);`
   2. From `consumeLoop`, upon hitting idle timeout. This is the primary source of noisy log messages.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #11565: [bugfix] Reduce noise level due to slow topics

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11565:
URL: https://github.com/apache/pinot/pull/11565#issuecomment-1713777826

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11565?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#11565](https://app.codecov.io/gh/apache/pinot/pull/11565?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (593df06) into [master](https://app.codecov.io/gh/apache/pinot/commit/bc63e0561030c77d818879bf1a6f8480417c1ea0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (bc63e05) will **decrease** coverage by `48.59%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #11565       +/-   ##
   =============================================
   - Coverage     63.07%   14.48%   -48.59%     
   + Complexity     1109      201      -908     
   =============================================
     Files          2325     2325               
     Lines        124772   124770        -2     
     Branches      19052    19052               
   =============================================
   - Hits          78695    18072    -60623     
   - Misses        40480   105170    +64690     
   + Partials       5597     1528     -4069     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-17](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.46%)` | :arrow_down: |
   | [java-20](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.59%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.59%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11565?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...manager/realtime/LLRealtimeSegmentDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/11565?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvTExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (-50.84%)` | :arrow_down: |
   
   ... and [1501 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11565/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] xiangfu0 merged pull request #11565: [logging] Reduce noise level due to slow topics

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #11565:
URL: https://github.com/apache/pinot/pull/11565


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org