You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ankitsultana (via GitHub)" <gi...@apache.org> on 2023/01/26 17:50:18 UTC
[GitHub] [pinot] ankitsultana opened a new issue, #10185: Restarting a Server with Partial Upsert Tables Leads to Tables in Bad State
ankitsultana opened a new issue, #10185:
URL: https://github.com/apache/pinot/issues/10185
We recently started using Partial Upsert tables for a use-case and started seeing this issue. We have a cluster with a few partial upsert tables with replication=1. If we restart a server in the cluster, all the tables (even offline/vanilla-realtime tables) go into Bad state.
In the server logs we see logs like the following:
```
2023/01/26 17:45:32.602 INFO [TableStateUtils] [HelixTaskExecutor-message_handle_thread_29] Find unloaded segment: my_great_table, table: my_great_table, expected: ONLINE, actual: OFFLINE
```
On taking a thread-dump I see as many threads as there are partial upsert tables in the cluster, all stuck in this loop (corresponding [PR](https://github.com/apache/pinot/pull/8923/files)):
https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/tablestate/TableStateUtils.java#L130
Sample thread-dump:
```
"HelixTaskExecutor-message_handle_thread_35" #116 daemon prio=5 os_prio=0 cpu=74100.46ms elapsed=3248.88s tid=0x00007eb8202b8000 nid=0xe9 waiting on condition [0x00007eb78b8f8000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@11.0.15/Native Method)
at org.apache.pinot.segment.local.utils.tablestate.TableStateUtils.waitForAllSegmentsLoaded(TableStateUtils.java:133)
at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:416)
- locked <0x00007ebbe43613b8> (a java.util.concurrent.atomic.AtomicBoolean)
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:189)
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:168)
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:83)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.15/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.15/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.15/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.15/Method.java:566)
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10185: Restarting a Server with Partial Upsert Tables Leads to Tables in Bad State
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10185:
URL: https://github.com/apache/pinot/issues/10185#issuecomment-1405741795
If it can be reproduced easily, can you check if the segment logged as unloaded is actually loaded or not?
For the other tables, can you check why it is showing as bad? You may compare the EV with IS to find the difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #10185: Restarting a Server with Partial Upsert Tables Leads to Tables in Bad State
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #10185:
URL: https://github.com/apache/pinot/issues/10185#issuecomment-1406922018
@Jackie-Jiang : Saw this issue again today and maybe the allSegmentsLoaded lock is an issue. The server was restarted around 10 hours ago from the moment I started debugging. This time most tables have recovered and only 1 partial upsert table is still having an issue.
For that table, there are 6 segments that should be in CONSUMING state as per IS that are in OFFLINE state. There's also 1 segment for the same table that should be in ONLINE state but is in ERROR state.
In the thread-dump, I see that around 5 threads are blocked, waiting to acquire the `allSegmentsLoaded` lock.
```
❯❯❯ cat 1.thdump | grep "0x00007f30e3b2ae68"
- waiting to lock <0x00007f30e3b2ae68> (a java.util.concurrent.atomic.AtomicBoolean)
- waiting to lock <0x00007f30e3b2ae68> (a java.util.concurrent.atomic.AtomicBoolean)
- waiting to lock <0x00007f30e3b2ae68> (a java.util.concurrent.atomic.AtomicBoolean)
- waiting to lock <0x00007f30e3b2ae68> (a java.util.concurrent.atomic.AtomicBoolean)
- waiting to lock <0x00007f30e3b2ae68> (a java.util.concurrent.atomic.AtomicBoolean)
- locked <0x00007f30e3b2ae68> (a java.util.concurrent.atomic.AtomicBoolean)
```
```
"HelixTaskExecutor-message_handle_thread_36" #117 daemon prio=5 os_prio=0 cpu=4243721.44ms elapsed=37170.30s tid=0x00007f2d1804d800 nid=0xda waiting on condition [0x00007f2c97dfb000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@11.0.15/Native Method)
at org.apache.pinot.segment.local.utils.tablestate.TableStateUtils.waitForAllSegmentsLoaded(TableStateUtils.java:133)
at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:416)
- locked <0x00007f30e3b2ae68> (a java.util.concurrent.atomic.AtomicBoolean)
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:189)
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:168)
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:83)
```
I do see a bunch of helix message handler threads are sitting idle (we have 50+ and only 6 are involved with that lock above, all presumably for the same table).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #10185: Restarting a Server with Partial Upsert Tables Leads to Tables in Bad State
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #10185: Restarting a Server with Partial Upsert Tables Leads to Tables in Bad State
URL: https://github.com/apache/pinot/issues/10185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10185: Restarting a Server with Partial Upsert Tables Leads to Tables in Bad State
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10185:
URL: https://github.com/apache/pinot/issues/10185#issuecomment-1405795170
I think I know the reason why it is also impacting regular tables.
For partial upsert table, we need to first load all sealed segments to restore the history before starting the consumption. Currently it blocks the state transition from OFFLINE -> CONSUMING. On each server, we allow limited state transitions to happen in parallel (40 by default). When multiple threads are blocked (worst case all 40 threads are blocked), other messages won't get processed.
We need to let the state transition pass, and block the consumption within the `LLRealtimeSegmentDataManager`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10185: Restarting a Server with Partial Upsert Tables Leads to Tables in Bad State
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10185:
URL: https://github.com/apache/pinot/issues/10185#issuecomment-1407110087
I see. This is great observation, and I believe there is another deadlock: Initially a segment is going through the OFFLINE -> CONSUMING state transition, then the segment is committed, so it becomes an ONLINE segment in IS. The `allSegmentsLoaded` check is waiting for the segment to become ONLINE, which will never happen because it is going through the OFFLINE -> CONSUMING state transition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org