You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "jadami10 (via GitHub)" <gi...@apache.org> on 2023/03/23 00:02:30 UTC
[GitHub] [pinot] jadami10 opened a new issue, #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
jadami10 opened a new issue, #10460:
URL: https://github.com/apache/pinot/issues/10460
We were working through an incident today where we lost a number of realtime servers. When new servers came up, they were waiting up to 45 minutes to catch up before `/health` would return `OK`. Because they were taking up to 30 minutes to consume up to realtime, we decided to force commit consuming segments so they could just download the data from S3.
For the most part, this worked great. But in one case, the server that was behind remained behind, and the other 2 healthy servers went from being 10 seconds of lag to reporting they were equally behind.
My hypothesis is the segment commit protocol picked the unhealthy/lagging server to be the committer. I'm confident we saw the lag spike after the force commit, so there is definitely an issue here. But I will spend some time validating the actual mechanism by which this happened tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1482063141
I agree with @jadami10 that there is a timeout of 3.3s to pick the winner with the best offset, so it may be useful to wait out that time.
I just don't recall the full discussion, so I will let @sajjad-moradi chime in here (he will be back Monday)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] navina commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1482036644
I think one of the use-cases that forcecommit was meant to solve the case where the ingestion is seems stuck. In such cases, force committing based on the first responsive replica rather than winning replica seems appropriate.
@jadami10 were you able to validate that forcecommit was the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1482010242
If a forced commit is in play, then we pick the first replica that the controller heard from, for a given segment. All replicas are given the opportunity to do so.
The idea to pick the first one is to NOT cause any additional delay (in terms of waiting for other replicas).
It may have happened that the one with largest lag was just too quick to respond :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1482021975
I believe we had a discussion on that front, and decided to add force commit the same as the criteria where the completion is due to numrows consumed. I don't recollect the reason here, but @sajjad-moradi may do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jadami10 commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1484278505
> To be exact, there is no timer that goes off in 3.3s. Whenever the next `segmentConsuming` call comes in, we check the time, and if it is more than 3.2s then we just pick a winner amongst the replicas that indicated intent to complete the segment (instead of waiting for all replicas).
That's true, but the server itself will only hold 3 seconds between commit attempts per https://github.com/apache/pinot/blob/989df641b7d29e18773a668663b0be894835d24f/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java#L62. It seems to be timed such that the server will try at most 3 times before a winner is picked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1483511426
Subbu is correct. The reason was to immediately kick off the force commit. Having said that, I agree that since the wait time to find out the server with the highest ingested offset is short - around 3s - we can modify the force commit to function like time-based commit scenarios and select the winner with the highest offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1482093713
> The `stopReason` seems to be `REASON_FORCE_COMMIT_MESSAGE_RECEIVED`, so it is identifiable. I wonder if the issue is this line,
>
> https://github.com/apache/pinot/blob/989df641b7d29e18773a668663b0be894835d24f/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java#L1174
>
> .
> The quickest fix (to this issue) would be to remove that line and let it wait for the rest of the servers. I know you mentioned we prefer to commit immediately with forceCommit, but the wait time isn't that long. `MAX_TIME_TO_PICK_WINNER_MS` seems to be hard coded to 3300ms.
This is indeed the case. I am less worried about the extent of the change, than the reason why we put it there in the first place. And yes, that was the exact reason we added another value to "stopReason" -- so that the completion manager can choose to take whatever action. All logic (when I look at it now) seems to suggest that we should move back to choosing the best offset behavior for this value of stopReason. But like I said, let us think a little bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jadami10 commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1482034980
The `stopReason` seems to be `REASON_FORCE_COMMIT_MESSAGE_RECEIVED`, so it is identifiable. I wonder if the issue is this line, https://github.com/apache/pinot/blob/989df641b7d29e18773a668663b0be894835d24f/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java#L1174.
The quickest fix (to this issue) would be to remove that line and let it wait for the rest of the servers. I know you mentioned we prefer to commit immediately with forceCommit, but the wait time isn't that long. `MAX_TIME_TO_PICK_WINNER_MS` seems to be hard coded to 3300ms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jadami10 commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1482056208
> @jadami10 were you able to validate that forcecommit was the issue?
yes, after looking through the logs of the winners, the server that was lagging was chosen as the winner for 5 partitions.
> I think one of the use-cases that forcecommit was meant to solve the case where the ingestion is seems stuck. In such cases, force committing based on the first responsive replica rather than winning replica seems appropriate.
Assuming my previous comment/reading of the code above is correct, when your ingestion is stuck and you call this, you will at most wait 3.3 seconds more seconds for a winner. In practice, it's likely minutes before you've noticed ingestion is stuck.
But in the case where 1 server is significantly behind (in our case 2 hours behind), accidentally picking it has pretty big ramifications. Now every server is 2 hours behind for those partitions.
There's also very few cases where the wait period is skipped (only row limit and end of partition reached). Even for time based or size based segment completion, the protocol still waits, and that doesn't seem to be a problem. I'll wait to hear from @sajjad-moradi, but I would be strongly in favor of undoing this part of the force commit feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
URL: https://github.com/apache/pinot/issues/10460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1483645250
To be exact, there is no timer that goes off in 3.3s. Whenever the next `segmentConsuming` call comes in, we check the time, and if it is more than 3.2s then we just pick a winner amongst the replicas that indicated intent to complete the segment (instead of waiting for all replicas).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1482016343
@mcvsubbu Thanks for the clarification! IMO we should still run the regular protocol to pick the server which consumes the most data. Force commit is just a trigger for the commit process. It might take longer to reach consensus, but will be more close to the regular commit behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10460: forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10460:
URL: https://github.com/apache/pinot/issues/10460#issuecomment-1482004723
@mcvsubbu @sajjad-moradi @npawar @navina In the consuming segment commit protocol, do we pick the candidate server that has the highest consuming offset to commit? If so, does the same logic apply to the force commit scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org