You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ankitsultana (via GitHub)" <gi...@apache.org> on 2025/01/15 22:47:36 UTC
[I] Critical Upsert Deadlock Condition Present in Pinot 1.1 [pinot]
ankitsultana opened a new issue, #14822:
URL: https://github.com/apache/pinot/issues/14822
We had been seeing frequent deadlocked threads on our Upsert enabled servers, and we were able to trace the root-cause to this commit: #12241.
At a high-level, the issue is that:
1. The commit introduced a new type of lock called "Upsert Lock".
2. There's no precedence defined between the "Segment Locks" and the "Upsert Locks". In some state transitions we may acquire Segment Lock first and then block on the Upsert Lock, and in others the order may be the opposite.
3. This can combine with the Lock Collision issue (see: #13632) that was fixed in July 2024, and it could be that the deadlock happens for two segments across two entirely different servers.
This impacts our internal Pinot version and also impacts Pinot 1.1 as far as I know. The Upsert Locks are now removed so I assume this would be fixed now. @Jackie-Jiang : would you be able to confirm this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upsert Deadlock Condition Present in Pinot 1.1 [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #14822:
URL: https://github.com/apache/pinot/issues/14822#issuecomment-2594113681
#12886 Refactored the message handling logic, and consolidated this lock into the regular segment lock. With #12886 and #13632 this problem should be solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upsert Deadlock Condition Present in Pinot 1.1 [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #14822: Upsert Deadlock Condition Present in Pinot 1.1
URL: https://github.com/apache/pinot/issues/14822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org