You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "jadami10 (via GitHub)" <gi...@apache.org> on 2023/09/07 03:57:10 UTC
[GitHub] [pinot] jadami10 opened a new pull request, #11534: do not emit lag once tabledatamanager shutdown
jadami10 opened a new pull request, #11534:
URL: https://github.com/apache/pinot/pull/11534
This is a fix for #11533.
First, we shutdown the ingestion delay first when the `RealtimeTableDataManger` is shutdown. This way the gauges are removed and stop reporting lag instead of waiting for all segments to get shutdown. Second, we stop emitting ingestion lag in `updateIngestionDelay` once the `RealtimeTableDataManger` is shutdown. This way, segments that have stopped consuming cannot keep updating ingestion delay.
I've tested this on a QA cluster consuming thousands of events per second. Before this, shutting down a server would show ~10 minutes of ingestion lag. After this, the ingestion lag stays stable and eventually stops reporting once all segments are destroyed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jugomezv commented on pull request #11534: do not emit lag once tabledatamanager shutdown
Posted by "jugomezv (via GitHub)" <gi...@apache.org>.
jugomezv commented on PR #11534:
URL: https://github.com/apache/pinot/pull/11534#issuecomment-1726625682
Thanks for the change @jadami10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jadami10 commented on pull request #11534: do not emit lag once tabledatamanager shutdown
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on PR #11534:
URL: https://github.com/apache/pinot/pull/11534#issuecomment-1709439252
@jugomezv, let me know what you think here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jugomezv commented on pull request #11534: do not emit lag once tabledatamanager shutdown
Posted by "jugomezv (via GitHub)" <gi...@apache.org>.
jugomezv commented on PR #11534:
URL: https://github.com/apache/pinot/pull/11534#issuecomment-1712098420
Also tagging @sajjad-moradi to take a look and make sure this may not cause other issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #11534: do not emit lag once tabledatamanager shutdown
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11534:
URL: https://github.com/apache/pinot/pull/11534
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11534: do not emit lag once tabledatamanager shutdown
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11534:
URL: https://github.com/apache/pinot/pull/11534#issuecomment-1709454232
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11534?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11534](https://app.codecov.io/gh/apache/pinot/pull/11534?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (5117d68) into [master](https://app.codecov.io/gh/apache/pinot/commit/be496bc763ac94660098c3a9e1f4a75167731789?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (be496bc) will **decrease** coverage by `48.54%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11534 +/- ##
=============================================
- Coverage 63.02% 14.48% -48.54%
+ Complexity 1106 201 -905
=============================================
Files 2320 2320
Lines 124660 124658 -2
Branches 19032 19033 +1
=============================================
- Hits 78566 18058 -60508
- Misses 40496 105070 +64574
+ Partials 5598 1530 -4068
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.51%)` | :arrow_down: |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.54%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.54%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11534?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...e/data/manager/realtime/IngestionDelayTracker.java](https://app.codecov.io/gh/apache/pinot/pull/11534?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvSW5nZXN0aW9uRGVsYXlUcmFja2VyLmphdmE=) | `0.00% <0.00%> (-62.77%)` | :arrow_down: |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/11534?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `0.00% <0.00%> (-32.02%)` | :arrow_down: |
... and [1493 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11534/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org