You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "nozjkoitop (via GitHub)" <gi...@apache.org> on 2025/06/09 11:24:27 UTC

[PR] Fix concurrent replace single phase tasks (druid)

nozjkoitop opened a new pull request, #18099:
URL: https://github.com/apache/druid/pull/18099

   ### Description
   
   #### The Issue
   REPLACE tasks (single-phase parallel index) that should delete rows were marked successful but did nothing - the row count stayed the same even though the thrownAway metric was positive.
   Logs showed that segments created by these tasks had a version of epoch (0).
   
   After some debugging, the flow appears to be:
   
   * When an existing lock can’t be obtained, the task [acquires a new one](https://github.com/apache/druid/blob/89cf9b263dc16238bd1493414826ea087f50d168/indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java#L867).
   * TaskLockAcquireAction [builds the lock request without preferredVersion](https://github.com/apache/druid/blob/89cf9b263dc16238bd1493414826ea087f50d168/indexing-service/src/main/java/org/apache/druid/indexing/common/actions/TimeChunkLockTryAcquireAction.java#L76).
   * The request [returns a version of epoch (0)](https://github.com/apache/druid/blob/89cf9b263dc16238bd1493414826ea087f50d168/indexing-service/src/main/java/org/apache/druid/indexing/common/actions/TaskLocks.java#L90) for [an APPEND lock](https://github.com/apache/druid/blob/89cf9b263dc16238bd1493414826ea087f50d168/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TimeChunkLockRequest.java#L118).
   * With concurrent locks enabled, the lock type is [always APPEND](https://github.com/apache/druid/blob/89cf9b263dc16238bd1493414826ea087f50d168/indexing-service/src/main/java/org/apache/druid/indexing/common/actions/TaskLocks.java#L158), even for REPLACE tasks.
   
   #### Possible fixes
   __1st option__ – Allow a REPLACE lock in [determineLockTypeForAppend](https://github.com/apache/druid/blob/12c1536e441011cd7d55dd05752d3567bd97fb16/indexing-service/src/main/java/org/apache/druid/indexing/common/actions/TaskLocks.java#L150).
   _Workaround-ish, since it needs an explicit lock type in the task context._
   
   _was_ 
   ```
   public static TaskLockType determineLockTypeForAppend(Map<String, Object> taskContext) {
     final boolean useConcurrentLocks = (boolean) taskContext.getOrDefault(
         Tasks.USE_CONCURRENT_LOCKS,
         Tasks.DEFAULT_USE_CONCURRENT_LOCKS
     );
     if (useConcurrentLocks) {
       return TaskLockType.APPEND;
     }
     final Object lockType = taskContext.get(Tasks.TASK_LOCK_TYPE);
     if (lockType == null) {
       final boolean useSharedLock = (boolean) taskContext.getOrDefault(Tasks.USE_SHARED_LOCK, false);
       return useSharedLock ? TaskLockType.SHARED : TaskLockType.EXCLUSIVE;
     } else {
       return TaskLockType.valueOf(lockType.toString());
     }
   }
   ```
   _became_ 
   ```
   public static TaskLockType determineLockTypeForAppend(Map<String, Object> taskContext) {
     final boolean useConcurrentLocks = (boolean) taskContext.getOrDefault(
         Tasks.USE_CONCURRENT_LOCKS,
         Tasks.DEFAULT_USE_CONCURRENT_LOCKS
     );
     final Object rawLock = taskContext.get(Tasks.TASK_LOCK_TYPE);
     TaskLockType lockType = (rawLock != null) ? TaskLockType.valueOf(rawLock.toString()) : null;
   
     if (useConcurrentLocks) {
       return (lockType == TaskLockType.REPLACE) ? TaskLockType.REPLACE : TaskLockType.APPEND;
     }
     if (lockType == null) {
       final boolean useSharedLock = (boolean) taskContext.getOrDefault(Tasks.USE_SHARED_LOCK, false);
       return useSharedLock ? TaskLockType.SHARED : TaskLockType.EXCLUSIVE;
     } else {
       return TaskLockType.valueOf(lockType.toString());
     }
   }
   
   ```
   __2nd option – The solution implemented in this PR.__
   It follows the TaskLocks comment about deduplicating determineLockTypeForAppend, making the logic cleaner and ensuring REPLACE tasks get the correct lock type and version.
   
   This PR has:
   
   - [X] been self-reviewed.
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [X] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955878763

   > Were you setting `maxNumConcurrentSubTasks` in the compaction config?
   
   in tuningConfig


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz merged PR #18099:
URL: https://github.com/apache/druid/pull/18099


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2956145558

   Created https://github.com/apache/druid/pull/18100 but that seems to try out some UTs with `useConcurrentLocks: true`.
   Seems to work fine. Issue is not reproducible with compaction yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-3542761178

   @nozjkoitop, apologies for the delay in repsonse.
   If you are still interested, would it please be possible to share sample ingestion specs along with the results with useConcurrentLocks set to true and false?
   Also thanks to you for the PR and to @kfaraz for looking into this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955727170

   Compaction works for me, not sure why yet, there is a difference in the processing i believe
   Row count is checked by simple COUNT(*), created segment with version `1970-01-01T00:00:00.000Z` is marked as unused right away


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955606656

   @nozjkoitop , could you please share the steps to reproduce the issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955836913

   @nozjkoitop , I have verified that tests do indeed fail.
   Must have been an oversight.
   
   Please apply the following diff to this PR to verify the tests:
   
   <details>
   <summary>Test patch to verify fix</summary>
   
   ```diff
   diff --git a/indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/SinglePhaseParallelIndexingTest.java b/indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/SinglePhaseParallelIndexingTest.java
   index fe573bf0f6..cc3cdffa74 100644
   --- a/indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/SinglePhaseParallelIndexingTest.java
   +++ b/indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/SinglePhaseParallelIndexingTest.java
   @@ -93,13 +93,13 @@ public class SinglePhaseParallelIndexingTest extends AbstractParallelIndexSuperv
      @Rule
      public ExpectedException expectedException = ExpectedException.none();
    
   -  @Parameterized.Parameters(name = "{0}, useInputFormatApi={1}, useSegmentCache={2}")
   +  @Parameterized.Parameters(name = "{0}, useInputFormatApi={1}, useSegmentCache={2}, useConcurrentLocks={3}")
      public static Iterable<Object[]> constructorFeeder()
      {
        return ImmutableList.of(
   -        new Object[]{LockGranularity.TIME_CHUNK, false, false},
   -        new Object[]{LockGranularity.TIME_CHUNK, true, true},
   -        new Object[]{LockGranularity.SEGMENT, true, false}
   +        new Object[]{LockGranularity.TIME_CHUNK, false, false, false},
   +        new Object[]{LockGranularity.TIME_CHUNK, true, true, true},
   +        new Object[]{LockGranularity.SEGMENT, true, false, true}
        );
      }
    
   @@ -108,18 +108,21 @@ public class SinglePhaseParallelIndexingTest extends AbstractParallelIndexSuperv
    
      private final LockGranularity lockGranularity;
      private final boolean useInputFormatApi;
   +  private final boolean useConcurrentLocks;
    
      private File inputDir;
    
      public SinglePhaseParallelIndexingTest(
          LockGranularity lockGranularity,
          boolean useInputFormatApi,
   -      boolean useSegmentMetadataCache
   +      boolean useSegmentMetadataCache,
   +      boolean useConcurrentLocks
      )
      {
        super(DEFAULT_TRANSIENT_TASK_FAILURE_RATE, DEFAULT_TRANSIENT_API_FAILURE_RATE, useSegmentMetadataCache);
        this.lockGranularity = lockGranularity;
        this.useInputFormatApi = useInputFormatApi;
   +    this.useConcurrentLocks = useConcurrentLocks;
      }
    
      @Before
   @@ -1004,7 +1007,7 @@ public class SinglePhaseParallelIndexingTest extends AbstractParallelIndexSuperv
            null,
            null,
            ingestionSpec,
   -        Collections.emptyMap()
   +        Map.of(Tasks.USE_CONCURRENT_LOCKS, useConcurrentLocks)
        );
      }
    
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955759262

   Hmm, this is weird. Trying to figure out why the single phase task was force using APPEND locks in the first place.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955662419

   Hmm, odd. @nozjkoitop , does it happen for compaction tasks too, since they are effectively replace jobs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955513186

   @AmatyaAvadhanula what do you think about this approach?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955776732

   Indeed, maybe @AmatyaAvadhanula, I see you handled most of the implementation. Could you share any insights when you have a moment?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955865448

   > Compaction works for me, not sure why yet, there is a difference in the processing i believe
   
   Were you setting `maxNumConcurrentSubTasks` in the compaction config?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2955659243

   > Does it just require running a replace job (i.e. `appendToExisting: false`) with `useConcurrentLocks: true` or are there other prerequisites too?
   
   maxNumConcurrentSubTasks should also be greater than 0 to trigger single phase task 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18099:
URL: https://github.com/apache/druid/pull/18099#discussion_r2135753840


##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/SinglePhaseParallelIndexTaskRunner.java:
##########
@@ -101,42 +99,25 @@ public class SinglePhaseParallelIndexTaskRunner extends ParallelIndexPhaseRunner
 
   private final ParallelIndexIngestionSpec ingestionSchema;
   private final SplittableInputSource<?> baseInputSource;
-  private CentralizedDatasourceSchemaConfig centralizedDatasourceSchemaConfig;
+  private final ParallelIndexSupervisorTask supervisorTask;
 
+  @VisibleForTesting

Review Comment:
   Annotation is not needed as this constructor is called outside tests too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fix concurrent replace single phase tasks (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18099:
URL: https://github.com/apache/druid/pull/18099#issuecomment-2956035534

   Yeah, not able to reproduce it for compaction tasks even with `maxNumConcurrentSubTasks: 2`.
   Maybe we can try it out on some tests similar to the patch shared above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org