You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "maytasm (via GitHub)" <gi...@apache.org> on 2025/01/16 05:29:45 UTC

[I] Race condition resulting in SegmentMissingException for query hitting realtime IncrementalSegment that was just announced. (druid)

maytasm opened a new issue, #17633:
URL: https://github.com/apache/druid/issues/17633

   Race condition resulting in SegmentMissingException for query hitting realtime IncrementalSegment that was just announced. 
   
   ### Affected Version
   
   31.0.1
   
   ### Description
   
   Wondering if this is a bug/race condition for query that is hitting realtime segments (IncrementalSegment) on the peon….
    At https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/segment/realtime/appenderator/StreamAppenderator.java#L514, we call segmentAnnouncer.announceSegment before adding the segment to sinkTimeline.add. Once the segment is announced, a query with time interval covering the segment will be sent to this peon (the broker will see that this announced segment is within the query interval and sends the query to the peon). However, all of this can happen before the sinkTimeline.add is done. This would result in the SegmentMissingException on the peon. 
   
   We saw this in our production cluster and try adding the below logs to confirm...
   
   StreamAppenderator#getOrCreateSink
   ```
    private Sink getOrCreateSink(final SegmentIdWithShardSpec identifier)
     {
       Sink retVal = sinks.get(identifier);
   
       if (retVal == null) {
         retVal = new Sink(
             identifier.getInterval(),
             schema,
             identifier.getShardSpec(),
             identifier.getVersion(),
             tuningConfig.getAppendableIndexSpec(),
             tuningConfig.getMaxRowsInMemory(),
             maxBytesTuningConfig,
             useMaxMemoryEstimates,
             null
         );
         bytesCurrentlyInMemory.addAndGet(calculateSinkMemoryInUsed(retVal));
         log.info("Announcing segment for interval=[%s] version=[%s], partition=[%s]", retVal.getInterval(), retVal.getVersion(), identifier.getShardSpec().getPartitionNum());
         try {
           segmentAnnouncer.announceSegment(retVal.getSegment());
         }
         catch (IOException e) {
           log.makeAlert(e, "Failed to announce new segment[%s]", schema.getDataSource())
              .addData("interval", retVal.getInterval())
              .emit();
         }
         log.info("Announced segment for interval=[%s] version=[%s], partition=[%s]", retVal.getInterval(), retVal.getVersion(), identifier.getShardSpec().getPartitionNum());
   
         sinks.put(identifier, retVal);
         metrics.setSinkCount(sinks.size());
         log.info("Creating sink time for interval=[%s] version=[%s], partition=[%s]", retVal.getInterval(), retVal.getVersion(), identifier.getShardSpec().getPartitionNum());
         sinkTimeline.add(retVal.getInterval(), retVal.getVersion(), identifier.getShardSpec().createChunk(retVal));
         log.info("Created sink time for interval=[%s] version=[%s], partition=[%s]", retVal.getInterval(), retVal.getVersion(), identifier.getShardSpec().getPartitionNum());
       }
   
       return retVal;
     }
   ```
   
   SinkQuerySegmentWalker#getQueryRunnerForSegments
   ```
       Iterable<QueryRunner<T>> perSegmentRunners = Iterables.transform(
           specs,
           newDescriptor -> {
             final SegmentDescriptor descriptor = newIdToBasePendingSegment.getOrDefault(newDescriptor, newDescriptor);
             final PartitionChunk<Sink> chunk = sinkTimeline.findChunk(
                 descriptor.getInterval(),
                 descriptor.getVersion(),
                 descriptor.getPartitionNumber()
             );
   
             if (chunk == null) {
               log.warn("Missing chunk for descriptor[%s]", descriptor);
               return new ReportTimelineMissingSegmentQueryRunner<>(descriptor);
             }
   ```
   
   This is then observed in the peon logs...
   ```
   2025-01-16T03:37:47,669 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Announcing segment for interval=[2025-01-16T03:00:00.000Z/2025-01-16T04:00:00.000Z] version=[2025-01-16T02:05:04.336Z], partition=[674]
   2025-01-16T03:37:47,685 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[realtimeqoe_log_count_2025-01-16T03:00:00.000Z_2025-01-16T04:00:00.000Z_2025-01-16T02:05:04.336Z_674] at new path[/xxx/_indexer-executor__default_tier_2025-01-16T03:37:47.685Z_b9eb03c7683347df8c77ab15abaf55ff0]
   2025-01-16T03:37:51,820 WARN [qtp1302132509-131[segmentMetadata_[realtimeqoe_log_count]_1619e80c-0b39-4e9b-9e94-cc2e643eabdc]] org.apache.druid.segment.realtime.appenderator.SinkQuerySegmentWalker - Missing chunk for descriptor[SegmentDescriptor{interval=2025-01-16T03:00:00.000Z/2025-01-16T04:00:00.000Z, version='2025-01-16T02:05:04.336Z', partitionNumber=674}]
   2025-01-16T03:37:52,046 WARN [qtp1302132509-135[segmentMetadata_[realtimeqoe_log_count]_1619e80c-0b39-4e9b-9e94-cc2e643eabdc]] org.apache.druid.segment.realtime.appenderator.SinkQuerySegmentWalker - Missing chunk for descriptor[SegmentDescriptor{interval=2025-01-16T03:00:00.000Z/2025-01-16T04:00:00.000Z, version='2025-01-16T02:05:04.336Z', partitionNumber=674}]
   2025-01-16T03:37:52,103 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Announced segment for interval=[2025-01-16T03:00:00.000Z/2025-01-16T04:00:00.000Z] version=[2025-01-16T02:05:04.336Z], partition=[674]
   2025-01-16T03:37:52,103 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Creating sink time for interval=[2025-01-16T03:00:00.000Z/2025-01-16T04:00:00.000Z] version=[2025-01-16T02:05:04.336Z], partition=[674]
   2025-01-16T03:37:52,116 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Created sink time for interval=[2025-01-16T03:00:00.000Z/2025-01-16T04:00:00.000Z] version=[2025-01-16T02:05:04.336Z], partition=[674]
   2025-01-16T03:40:07,302 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Persisted rows[150,000] and (estimated) bytes[654,551,224]
   2025-01-16T03:40:11,093 INFO [[index_kafka_realtimeqoe_log_count_465e92bb00f87b1_nhdmljci]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data for segment[realtimeqoe_log_count_2025-01-16T03:00:00.000Z_2025-01-16T04:00:00.000Z_2025-01-16T02:05:04.336Z_674] spill[0] to disk in [3,791] ms (150,000 rows).
   
   ```
   
   Maybe we should add to the sinkTimeline before announcing? 
   Also, the retry for SegmentMissingException doesn't really help as there is no delay/backoff and happen before the segment can be added to the sinkTimeline


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Race condition resulting in SegmentMissingException for query hitting realtime IncrementalSegment that was just announced. (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm closed issue #17633: Race condition resulting in SegmentMissingException for query hitting realtime IncrementalSegment that was just announced.
URL: https://github.com/apache/druid/issues/17633


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Race condition resulting in SegmentMissingException for query hitting realtime IncrementalSegment that was just announced. (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm closed issue #17633: Race condition resulting in SegmentMissingException for query hitting realtime IncrementalSegment that was just announced.
URL: https://github.com/apache/druid/issues/17633


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org