You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "vivek807 (via GitHub)" <gi...@apache.org> on 2026/06/19 11:22:21 UTC

[PR] fix: Fix invalid time comparison (druid)

vivek807 opened a new pull request, #19603:
URL: https://github.com/apache/druid/pull/19603

   Fixes #19602.
   
   ### Description
   
   `forceOrWaitOngoingDatabasePoll` in `SqlSegmentsMetadataManager` checks whether a concurrent `OnDemandDatabasePoll` that completed while the calling thread waited for the write lock makes a new poll unnecessary. This check was always false due to a clock mismatch, causing an extra database poll on every contended call.
   
   #### Fixed invalid clock comparison in `forceOrWaitOngoingDatabasePoll`
   
   `OnDemandDatabasePoll.initiationTimeNanos` is assigned via `System.nanoTime()`, which is a monotonic clock with an arbitrary JVM-internal reference point.
   The comparison value was constructed as:
   
   ```java
   long checkStartTimeNanos = TimeUnit.MILLISECONDS.toNanos(checkStartTime);
   // checkStartTime = System.currentTimeMillis()
   ```
   
   System.currentTimeMillis() returns milliseconds since the Unix epoch (~1.7 × 10¹² ms). Converting to nanoseconds yields ~1.7 × 10²¹ ns, which is orders of magnitude larger than any System.nanoTime() value. The condition `initiationTimeNanos > checkStartTimeNanos` was therefore always false, so an in-progress on-demand poll was never recognised as sufficient.
   <!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
   
   <!-- If you are a committer, follow the PR action item checklist for committers:
   https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
   
   This PR has:
   
   - [x] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #19603:
URL: https://github.com/apache/druid/pull/19603#issuecomment-5151939927

   Thanks for the changes, @vivek807 !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19603:
URL: https://github.com/apache/druid/pull/19603#discussion_r3442409940


##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -569,7 +570,6 @@ void forceOrWaitOngoingDatabasePoll()
         }
         // Verify if there was a on-demand poll completed while we were waiting for the lock
         if (latestDatabasePoll instanceof OnDemandDatabasePoll) {
-          long checkStartTimeNanos = TimeUnit.MILLISECONDS.toNanos(checkStartTime);
           OnDemandDatabasePoll latestOnDemandPoll = (OnDemandDatabasePoll) latestDatabasePoll;
           if (latestOnDemandPoll.initiationTimeNanos > checkStartTimeNanos) {

Review Comment:
   [P2] Check poll success before reusing a contended on-demand poll
   
   Now that this nanoTime comparison can be true, a caller queued behind another on-demand poll can return without checking whether that poll succeeded. If the first poll throws, doOnDemandPoll completes pollCompletionFuture exceptionally and releases the write lock; any caller whose checkStartTimeNanos was before that poll started will hit this branch and return the stale dataSourcesSnapshot instead of retrying or propagating the failure. The existing catch says unsuccessful latest polls should trigger a new poll, so this branch should verify pollCompletionFuture, for example with Futures.getUnchecked(latestOnDemandPoll.pollCompletionFuture), before returning.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19603:
URL: https://github.com/apache/druid/pull/19603#discussion_r3499360854


##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -556,22 +556,22 @@ boolean useLatestSnapshotIfWithinDelay()
   @VisibleForTesting
   void forceOrWaitOngoingDatabasePoll()
   {
-    long checkStartTime = System.currentTimeMillis();
+    long checkStartTimeNanos = System.nanoTime();
     ReentrantReadWriteLock.WriteLock lock = startStopPollLock.writeLock();
     lock.lock();
     try {
       DatabasePoll latestDatabasePoll = this.latestDatabasePoll;
       try {
         //Verify if there was a periodic poll completed while we were waiting for the lock
         if (latestDatabasePoll instanceof PeriodicDatabasePoll
-            && ((PeriodicDatabasePoll) latestDatabasePoll).lastPollStartTimestampInMs > checkStartTime) {
+            && ((PeriodicDatabasePoll) latestDatabasePoll).lastPollStartTimestampInNanos > checkStartTimeNanos) {
           return;
         }
-        // Verify if there was a on-demand poll completed while we were waiting for the lock
+        // Verify if there was an on-demand poll completed while we were waiting for the lock
         if (latestDatabasePoll instanceof OnDemandDatabasePoll) {
-          long checkStartTimeNanos = TimeUnit.MILLISECONDS.toNanos(checkStartTime);
           OnDemandDatabasePoll latestOnDemandPoll = (OnDemandDatabasePoll) latestDatabasePoll;
           if (latestOnDemandPoll.initiationTimeNanos > checkStartTimeNanos) {

Review Comment:
   [P2] Contended force calls still miss the poll they waited for
   
   This still compares the existing on-demand poll's initiation time with the caller's check start. In the common contended path, thread A creates the OnDemandDatabasePoll and holds the write lock while polling; thread B enters forceOrWaitOngoingDatabasePoll after that initiation, waits for the same write lock until A's poll completes, then sees initiationTimeNanos <= checkStartTimeNanos and starts another database poll. That means the PR still serializes duplicate forced polls for callers that arrived during the actual poll, even though the previous poll completed after their call. Track and compare completion time, or otherwise record that the caller waited behind the in-progress on-demand poll, before returning/forcing a new poll.



##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -90,7 +90,7 @@ static class PeriodicDatabasePoll implements DatabasePoll
      * leadership changes.
      */
     final CompletableFuture<Void> firstPollCompletionFuture = new CompletableFuture<>();
-    long lastPollStartTimestampInMs = -1;
+    long lastPollStartTimestampInNanos = -1;

Review Comment:
   [P2] Do not use -1 as a nanoTime sentinel
   
   This timestamp is now compared directly with System.nanoTime() readings, but System.nanoTime() has an arbitrary origin and is allowed to return negative values. If the manager has installed a new PeriodicDatabasePoll but its first task has not set this field yet, lastPollStartTimestampInNanos remains -1; on a JVM where the caller's checkStartTimeNanos is less than -1, forceOrWaitOngoingDatabasePoll will treat that unstarted poll as newer and return without polling or waiting. Use Long.MIN_VALUE, an explicit initialized flag, or a nullable/completion timestamp instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz merged PR #19603:
URL: https://github.com/apache/druid/pull/19603


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19603:
URL: https://github.com/apache/druid/pull/19603#discussion_r3510703814


##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -90,7 +90,7 @@ static class PeriodicDatabasePoll implements DatabasePoll
      * leadership changes.
      */
     final CompletableFuture<Void> firstPollCompletionFuture = new CompletableFuture<>();
-    long lastPollStartTimestampInMs = -1;
+    long lastPollStartTimestampInNanos = -1;

Review Comment:
   Yeah, I suppose `null` would be a better choice for this now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "vivek807 (via GitHub)" <gi...@apache.org>.
vivek807 commented on PR #19603:
URL: https://github.com/apache/druid/pull/19603#issuecomment-4796964688

   > @vivek807 Thanks for flagging this issue. You are right, the values are not comparable. I'm thinking that maybe instead we should standardize the timestamps in OnDemandDatabasePoll and PeriodicDatabasePoll to use the same format (i.e. both use System.nanoTime() or both use System.currentTimeMillis()). This can avoid similar confusion / bug like this in the future. Wdyt? Thanks!
   
   Yes, this will avoid confusion in future. Let me refactor it to use same type of clock here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "maytasm (via GitHub)" <gi...@apache.org>.
maytasm commented on PR #19603:
URL: https://github.com/apache/druid/pull/19603#issuecomment-4771613656

   @vivek807 
   Thanks for flagging this issue. You are right, the values are not comparable. 
   I'm thinking that maybe instead we should standardize the timestamps in OnDemandDatabasePoll and PeriodicDatabasePoll to use the same format (i.e. both use System.nanoTime() or both use System.currentTimeMillis()). This can avoid similar confusion / bug like this in the future. Wdyt? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19603:
URL: https://github.com/apache/druid/pull/19603#discussion_r3489910141


##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -567,11 +567,11 @@ void forceOrWaitOngoingDatabasePoll()
             && ((PeriodicDatabasePoll) latestDatabasePoll).lastPollStartTimestampInMs > checkStartTime) {
           return;
         }
-        // Verify if there was a on-demand poll completed while we were waiting for the lock
+        // Verify if there was an on-demand poll completed while we were waiting for the lock
         if (latestDatabasePoll instanceof OnDemandDatabasePoll) {
-          long checkStartTimeNanos = TimeUnit.MILLISECONDS.toNanos(checkStartTime);
           OnDemandDatabasePoll latestOnDemandPoll = (OnDemandDatabasePoll) latestDatabasePoll;
-          if (latestOnDemandPoll.initiationTimeNanos > checkStartTimeNanos) {
+          if (latestOnDemandPoll.initiationTimeMillis > checkStartTime) {
+            Futures.getUnchecked(latestOnDemandPoll.pollCompletionFuture);

Review Comment:
   Is this really needed?



##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -100,12 +100,12 @@ static class PeriodicDatabasePoll implements DatabasePoll
   @VisibleForTesting
   static class OnDemandDatabasePoll implements DatabasePoll
   {
-    final long initiationTimeNanos = System.nanoTime();
+    final long initiationTimeMillis = System.currentTimeMillis();

Review Comment:
   Let's stick to using nanos since it is more reliably monotonic.



##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -567,11 +567,11 @@ void forceOrWaitOngoingDatabasePoll()
             && ((PeriodicDatabasePoll) latestDatabasePoll).lastPollStartTimestampInMs > checkStartTime) {
           return;
         }
-        // Verify if there was a on-demand poll completed while we were waiting for the lock
+        // Verify if there was an on-demand poll completed while we were waiting for the lock
         if (latestDatabasePoll instanceof OnDemandDatabasePoll) {
-          long checkStartTimeNanos = TimeUnit.MILLISECONDS.toNanos(checkStartTime);

Review Comment:
   We should change this line to simply be:
   
   ```java
   long checkStartTimeNanos = System.nanoTime();
   ```
   
   We should also change `PeriodicDatabasePoll.lastPollStartTimestampInMs` to be `lastPollStartTimestampInNanos`.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19603:
URL: https://github.com/apache/druid/pull/19603#discussion_r3452104533


##########
server/src/test/java/org/apache/druid/metadata/SqlSegmentsMetadataManagerTest.java:
##########
@@ -231,6 +233,51 @@ public void testPollPeriodicallyAndOnDemandInterleave() throws Exception
     );
   }
 
+  @Test(timeout = 30_000)
+  public void testForceOrWait_blocksUntilConcurrentPollCompletes() throws Exception
+  {
+    Field lockField = SqlSegmentsMetadataManager.class.getDeclaredField("startStopPollLock");
+    lockField.setAccessible(true);
+    ReentrantReadWriteLock rwLock = (ReentrantReadWriteLock) lockField.get(sqlSegmentsMetadataManager);
+
+    Field pollField = SqlSegmentsMetadataManager.class.getDeclaredField("latestDatabasePoll");
+    pollField.setAccessible(true);
+
+    Thread caller = new Thread(() -> sqlSegmentsMetadataManager.forceOrWaitOngoingDatabasePoll());
+    caller.setDaemon(true);
+
+    SqlSegmentsMetadataManager.OnDemandDatabasePoll concurrentPoll;
+
+    rwLock.writeLock().lock();
+    try {
+      caller.start();
+      // Allow the caller to capture checkStartTimeNanos and block on the write lock.
+      Thread.sleep(100);

Review Comment:
   [P3] Replace timing sleep with deterministic thread coordination
   
   The new regression test depends on the caller thread being scheduled within this fixed 100 ms window so it captures checkStartTimeNanos before concurrentPoll is created. On a slow or overloaded CI worker, the caller can start after the sleep, making concurrentPoll.initiationTimeNanos <= checkStartTimeNanos; then the production branch under test is skipped and the caller may run its own poll or return early, causing a flaky failure or a false pass. Please coordinate on the lock queue/state, e.g. wait until the caller is queued on the write lock before creating concurrentPoll, instead of relying on sleep timing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: Fix invalid time comparison (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19603:
URL: https://github.com/apache/druid/pull/19603#discussion_r3481290021


##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -100,12 +100,12 @@ static class PeriodicDatabasePoll implements DatabasePoll
   @VisibleForTesting
   static class OnDemandDatabasePoll implements DatabasePoll
   {
-    final long initiationTimeNanos = System.nanoTime();
+    final long initiationTimeMillis = System.currentTimeMillis();
     final CompletableFuture<Void> pollCompletionFuture = new CompletableFuture<>();
 
-    long nanosElapsedFromInitiation()
+    long millisElapsedFromInitiation()
     {
-      return System.nanoTime() - initiationTimeNanos;
+      return System.currentTimeMillis() - initiationTimeMillis;

Review Comment:
   [P2] Use monotonic time for freshness elapsed checks
   
   `millisElapsedFromInitiation()` now computes elapsed time from `System.currentTimeMillis()`. If the wall clock moves backward after an on-demand poll starts, this value can become negative or too small, so `useLatestSnapshotIfWithinDelay()` can keep accepting a stale snapshot and the periodic poll task can sleep longer than `periodicPollDelay`. Keep a monotonic `nanoTime()` value for elapsed/freshness calculations and use a separate wall-clock timestamp only where this code compares against wall-clock poll start times.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org