[GitHub] [pinot] ankitsultana commented on issue #10552: Partial Upsert table ingestion stops after server restarts

["ankitsultana (via GitHub)" <gi...@apache.org>]

ankitsultana commented on issue #10552:
URL: https://github.com/apache/pinot/issues/10552#issuecomment-1512529716

   @Jackie-Jiang : We saw this again today for one of our clusters. I saw this:
   
   1. In the logs there was a message for the 235th segment of the a Kafka partition:
   
   ```
   2023-04-18 06:31:11.871 [my_table__224__236__20230418T0520Z] INFO  o.a.pinot.segment.local.utils.tablestate.TableStateUtils  - Found 1 unloaded segments: [my_table__64__235__20230417T2021Z] for table: my_table_REALTIME
   ```
   
   2. In the thread-dump I saw that the consumeLoop had started consuming for the 236th segment already and was waiting on all segments to be loaded:
   
   ```
   "my_table_v3__64__236__20230418T0511Z" #1992 daemon prio=5 os_prio=0 cpu=1911.79ms elapsed=4383.52s tid=0x00007efb10016800 nid=0x967 waiting on condition  [0x00007ee0714fa000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
           at java.lang.Thread.sleep(java.base@11.0.15/Native Method)
           at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:406)
           at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:647)
           at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)
   ```
   
   This seems like a deadlock. We can't even reset segments to get out of this since it gets stuck in acquiring the `_partitionGroupConsumerSemaphore`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org