You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "sylph-eu (via GitHub)" <gi...@apache.org> on 2023/09/11 11:45:48 UTC
[GitHub] [pinot] sylph-eu opened a new pull request, #11565: [bugfix] Reduce noise level due to slow topics
sylph-eu opened a new pull request, #11565:
URL: https://github.com/apache/pinot/pull/11565
Our deployment of Pinot has plenty of "slow" REALTIME table for which no new data are produced for several minutes in a row. As the result, stream consumers are regularly recreated with the following WARN message
```
Recreating stream consumer for topic partition <kafka-topic>-<partition>, reason: Total idle time: 183623 ms exceeded idle timeout: 180000 ms
```
I'd like to reduce the level of this message to drop plenty of noisy WARN+ messages that pollutes our logging system.
Speaking of the actual usage, this message inside `recreateStreamConsumer` being called from two places:
1. From `handleTransientStreamErrors`, when max consecutive error count exceeds the threshold. In this case there's another warning message `_segmentLogger
.warn("Stream transient exception when fetching messages, retrying (count={})", _consecutiveErrorCount, e);`
2. From `consumeLoop`, upon hitting idle timeout. This is the primary source of noisy log messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11565: [bugfix] Reduce noise level due to slow topics
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11565:
URL: https://github.com/apache/pinot/pull/11565#issuecomment-1713777826
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11565?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11565](https://app.codecov.io/gh/apache/pinot/pull/11565?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (593df06) into [master](https://app.codecov.io/gh/apache/pinot/commit/bc63e0561030c77d818879bf1a6f8480417c1ea0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (bc63e05) will **decrease** coverage by `48.59%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11565 +/- ##
=============================================
- Coverage 63.07% 14.48% -48.59%
+ Complexity 1109 201 -908
=============================================
Files 2325 2325
Lines 124772 124770 -2
Branches 19052 19052
=============================================
- Hits 78695 18072 -60623
- Misses 40480 105170 +64690
+ Partials 5597 1528 -4069
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.46%)` | :arrow_down: |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.59%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.59%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11565/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11565?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...manager/realtime/LLRealtimeSegmentDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/11565?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvTExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (-50.84%)` | :arrow_down: |
... and [1501 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11565/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] xiangfu0 merged pull request #11565: [logging] Reduce noise level due to slow topics
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #11565:
URL: https://github.com/apache/pinot/pull/11565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org