Re: [PR] feat: Reset offsets and backfill (druid)

["abhishekrb19 (via GitHub)" <gi...@apache.org>]

abhishekrb19 commented on code in PR #19191:
URL: https://github.com/apache/druid/pull/19191#discussion_r3052026045


##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/supervisor/KafkaSupervisor.java:
##########
@@ -265,6 +267,130 @@ protected List<SeekableStreamIndexTask<KafkaTopicPartition, Long, KafkaRecordEnt
     return taskList;
   }
 
+  @Override
+  public void submitBackfillTask(
+      Map<KafkaTopicPartition, Long> startOffsets,
+      Map<KafkaTopicPartition, Long> endOffsets
+  )
+  {
+    if (startOffsets == null || startOffsets.isEmpty() || endOffsets == null || endOffsets.isEmpty()) {
+      log.info("No offsets to backfill, skipping backfill task submission");
+      return;
+    }
+
+    try {
+      String backfillSupervisorId = spec.getSpec().getDataSchema().getDataSource() + "_backfill";
+
+      // Get the backfillTaskCount from config
+      int backfillTaskCount = spec.getSpec().getIOConfig().getBackfillTaskCount();

Review Comment:
   Is there a reason to include this in the supervisor's `ioConfig`? It feels like a property that should be supplied by the operator when invoking the API, rather than being defined at the supervisor's `ioConfig`. It could instead default to `taskCount / 2` (or a similar value) as part of the API invocation.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorResource.java:
##########
@@ -592,9 +592,16 @@ public Response specGetHistory(
   @Path("/{id}/reset")
   @Produces(MediaType.APPLICATION_JSON)
   @ResourceFilters(SupervisorResourceFilter.class)
-  public Response reset(@PathParam("id") final String id)
+  public Response reset(
+      @PathParam("id") final String id,
+      @QueryParam("backfill") Boolean backfill

Review Comment:
   IMO  it would be cleaner to introduce a new endpoint `/{id}/resetAndBackfill` especially if we’re adding more backfill-specific properties



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org