You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/11/10 23:46:46 UTC
[GitHub] [pinot] dang-stripe opened a new issue, #9784: Query freshness lag after running rebalance w/ includeConsuming=false
dang-stripe opened a new issue, #9784:
URL: https://github.com/apache/pinot/issues/9784
We're currently doing this for realtime table rebalances to minimize data ingestion lag:
1. Rebalance table w/ includeConsuming=false - instance partition map is updated w/ new assignments, but no new consuming segments are created
2. forceCommit - ensure all consuming segments are sealed and new consuming segments created w/ updated assignment
3. Rebalance table again w/ includeConsuming=false - ensure lingering consuming segments are moved to correct hosts
We've noticed large spikes (within bounds of the segment flush time threshold) in the broker's freshness_lag_ms metric for realtime tables after step 1 that don't go away until step 2 is complete. We suspect this might either be related to rebalance w/ includeConsuming=false or the metric may be reporting incorrectly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Query freshness lag spikes after running rebalance w/ includeConsuming=false [pinot]
Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe closed issue #9784: Query freshness lag spikes after running rebalance w/ includeConsuming=false
URL: https://github.com/apache/pinot/issues/9784
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] dang-stripe commented on issue #9784: Query freshness lag spikes after running rebalance w/ includeConsuming=false
Posted by GitBox <gi...@apache.org>.
dang-stripe commented on issue #9784:
URL: https://github.com/apache/pinot/issues/9784#issuecomment-1311053368
Alternatively, we've tried this - this seems to avoid the issue above while having a slightly longer ingestion delay.
1. Force commit the table
2. Rebalance w/ includeConsuming=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9784: Query freshness lag spikes after running rebalance w/ includeConsuming=false
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9784:
URL: https://github.com/apache/pinot/issues/9784#issuecomment-1332863780
Checked the code and confirmed the behavior: when the committed segments are moved to a different server, when that server is queried, we can only use the max end time of the committed segments as the freshness time, which leads to this freshness lag spike. After the consuming segment is moved, the freshness lag will get back to normal.
This issue also applies to the tables that have committed segments relocated to a different server. We should find out a way to handle it properly.
Tl;dr: the spike is from freshness mis-calculation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9784: Query freshness lag spikes after running rebalance w/ includeConsuming=false
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9784:
URL: https://github.com/apache/pinot/issues/9784#issuecomment-1320877569
Interesting. I think it is because the metric is incorrect when the consuming segment and committed segments are in different servers, and the freshness lag for the server with committed segments spikes because there is no consuming segment on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Query freshness lag spikes after running rebalance w/ includeConsuming=false [pinot]
Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on issue #9784:
URL: https://github.com/apache/pinot/issues/9784#issuecomment-1853293395
Closing this as a duplicate of https://github.com/apache/pinot/issues/11448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org