You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ankitsultana (via GitHub)" <gi...@apache.org> on 2023/07/21 01:32:08 UTC
[GitHub] [pinot] ankitsultana opened a new issue, #11145: Lock Contention in Controller During Realtime Segment Commit Events
ankitsultana opened a new issue, #11145:
URL: https://github.com/apache/pinot/issues/11145
We are seeing an issue with one of our high ingestion throughput tables where ingestion lag continues to increase because of a table-level lock in Pinot controller.
https://github.com/apache/pinot/blob/723b764bc91275c0b8361d3f9135f151b6404c39/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L594
@Jackie-Jiang I think we briefly discussed this yesterday. I don't have enough context about this, but one potential solution could be to add some jitter in the number of docs in a segment at partition level so these events arrive at slightly different times. But I don't think that is tenable and we may need a more proper fix.
Also, there is a lock being taken in logback which is causing a lot of contention (there were 10 threads blocked in a sample thread dump I took). That seems like a simpler fix which we may wanna do anyways.
```
...
"grizzly-http-server-435" #625 prio=5 os_prio=0 cpu=16853811.42ms elapsed=7356619.65s tid=0x00007f0283f80000 nid=0x2d1 waiting for monitor entry [0x00007edb0c584000]
java.lang.Thread.State: BLOCKED (on object monitor)
at ch.qos.logback.classic.LoggerContext.getLogger(LoggerContext.java:151)
- waiting to lock <0x00007ee26209cf80> (a ch.qos.logback.classic.Logger)
at ch.qos.logback.classic.LoggerContext.getLogger(LoggerContext.java:53)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363)
at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.<init>(SegmentCompletionManager.java:437)
at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.fsmInHolding(SegmentCompletionManager.java:399)
at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.lookupOrCreateFsm(SegmentCompletionManager.java:162)
at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.segmentConsumed(SegmentCompletionManager.java:197)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #11145: Lock Contention in Controller During Realtime Segment Commit Events
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #11145:
URL: https://github.com/apache/pinot/issues/11145#issuecomment-1734641682
Just for clarity, our particular issue was due to the [IS update lock](https://github.com/apache/pinot/blob/723b764bc91275c0b8361d3f9135f151b6404c39/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L594)
The PR #11679 removes the FSM lock: did you find that to be the bottleneck for some use-case?
The FSM lock isn't technically global and uses striping (num-locks=20) with segmentName as the hash key. Also the ops done under the lock should be relatively quite quick, so wondering if there'll be much improvement (since IS update will continue to be the bottleneck).
It's been a while since I looked into this so I might need to take a deeper look if more context is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11145: Lock Contention in Controller During Realtime Segment Commit Events
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11145:
URL: https://github.com/apache/pinot/issues/11145#issuecomment-1734674857
@ankitsultana I'll use this ticket to track a series of improvements. The thread dump you took above is actually blocking on creating the FSM, which is solved with #11679.
I referred it as global because the locks are actually cross table, and segment from different table can block each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] priyen commented on issue #11145: Lock Contention in Controller During Realtime Segment Commit Events
Posted by "priyen (via GitHub)" <gi...@apache.org>.
priyen commented on issue #11145:
URL: https://github.com/apache/pinot/issues/11145#issuecomment-1645682828
@Jackie-Jiang yes, this and force commit's should manifest similarly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11145: Lock Contention in Controller During Realtime Segment Commit Events
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11145:
URL: https://github.com/apache/pinot/issues/11145#issuecomment-1645185035
@jadami10 @priyen This is similar to the issue you run into
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11145: Lock Contention in Controller During Realtime Segment Commit Events
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11145:
URL: https://github.com/apache/pinot/issues/11145#issuecomment-1734540542
#11679 removes the global lock on FSM creation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org