[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7927: Fix realtime ingestion when an entire batch of messages is filtered out

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on a change in pull request #7927:
URL: https://github.com/apache/pinot/pull/7927#discussion_r772591383



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
##########
@@ -294,15 +300,17 @@ public void deleteSegmentFile() {
   private boolean endCriteriaReached() {
     Preconditions.checkState(_state.shouldConsume(), "Incorrect state %s", _state);
     long now = now();
+    long consumeEndTime = _consumeEndTime;
     switch (_state) {
       case INITIAL_CONSUMING:
         // The segment has been created, and we have not posted a segmentConsumed() message on the controller yet.
         // We need to consume as much data as available, until we have either reached the max number of rows or
         // the max time we are allowed to consume.
-        if (now >= _consumeEndTime) {
+        if (now >= consumeEndTime) {
           if (_realtimeSegment.getNumDocsIndexed() == 0) {
             _segmentLogger.info("No events came in, extending time by {} hours", TIME_EXTENSION_ON_EMPTY_SEGMENT_HOURS);
-            _consumeEndTime += TimeUnit.HOURS.toMillis(TIME_EXTENSION_ON_EMPTY_SEGMENT_HOURS);
+            CONSUME_END_TIME_UPDATER.compareAndSet(this, consumeEndTime, consumeEndTime
+                + TimeUnit.HOURS.toMillis(TIME_EXTENSION_ON_EMPTY_SEGMENT_HOURS));

Review comment:
       There is a single consuming thread, so it should not have concurrency issue. I'd suggest keeping the existing code to avoid the unnecessary overhead of using updater. We can add annotation to disable the IDE warning on updating volatile fields and comment about the single thread consuming behavior.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org