[GitHub] [incubator-druid] teeram commented on issue #8663: Kafka indexing service duplicate entry exception in druid_pendingSegments

[GitBox <gi...@apache.org>]

teeram commented on issue #8663: Kafka indexing service duplicate entry exception in druid_pendingSegments
URL: https://github.com/apache/incubator-druid/issues/8663#issuecomment-541202721
 
 
   @jihoonson, thank you so much for looking into this.
   
   1. I would say this error occurs roughly 33% of the time I repeat the process described above.
   2. I'm not explicitly updating the `druid_pendingSegments` or `druid_segments` tables between step 2 and 4. I generally don't touch anything in metadata storage and always try to work within the standard Druid api's. I don't trust myself to not break something :) But I do occasionally peek in there when I'm doing some debugging.
   
   So I wrote a small utility that will suspend a supervisor and drop various segments for a given time period from that data source before resuming supervisor. It does wait for tasks to finish and segments to be fully unloaded before continuing on to the next steps. I updated this utility to go and delete any entries for a given data source in the `druid_pendingSegments` table before resuming the supervisor. This seems to prevent the duplicate key error from occurring.
   
   Any chance there were some changes between 0.14.1 and 0.16.0 to the way the `druid_pendingSegments` table is manipulated? I upgraded straight from 0.14.1 to 0.16.0 and skipped some of the 0.15.X versions. Maybe that was a bad move on my part? I don't believe I had this issue in 0.14.1 so I'm guessing something changed between that release and 0.16.0. Thanks again for all your help!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org