Re: [PR] feat: Add Supervisor UI component to view timeline of rules applied by a cascading reindexing supervisor (druid)

["FrankChen021 (via GitHub)" <gi...@apache.org>]

FrankChen021 commented on code in PR #19051:
URL: https://github.com/apache/druid/pull/19051#discussion_r3141586117


##########
indexing-service/src/main/java/org/apache/druid/indexing/compact/CascadingReindexingTemplate.java:
##########
@@ -262,6 +288,121 @@ private static boolean intervalEndsAfter(Interval interval, DateTime boundary)
     return interval.getEnd().isAfter(boundary);
   }
 
+  /**
+   * Generates a timeline view showing the search intervals and their associated reindexing
+   * configurations. This is useful for operators to understand how rules are applied across
+   * different time periods and to preview the effects of rule changes before they are applied.
+   *
+   * @param referenceTime the reference time to use for computing rule periods (typically DateTime.now())
+   * @return a view of the reindexing timeline with intervals and their configs
+   */
+  public ReindexingTimelineView getReindexingTimelineView(DateTime referenceTime)
+  {
+    if (!ruleProvider.isReady()) {
+      LOG.info(
+          "Rule provider [%s] is not ready, returning empty timeline for dataSource[%s]",
+          ruleProvider.getType(),
+          dataSource
+      );
+      return new ReindexingTimelineView(dataSource, referenceTime, null, Collections.emptyList(), null);
+    }
+
+    List<IntervalPartitioningInfo> searchIntervals;
+    try {
+      searchIntervals = generateAlignedSearchIntervals(referenceTime);
+    }
+    catch (SegmentGranularityTimelineValidationException e) {
+      return createValidationErrorView(
+          e,
+          referenceTime,
+          "INVALID_GRANULARITY_TIMELINE",
+          e.getOlderInterval().toString(),
+          e.getOlderGranularity().toString(),
+          e.getNewerInterval().toString(),
+          e.getNewerGranularity().toString()
+      );
+    }
+    catch (IAE e) {
+      return createValidationErrorView(
+          e,
+          referenceTime,
+          "VALIDATION_ERROR",
+          null,
+          null,
+          null,
+          null
+      );
+    }
+
+    if (searchIntervals.isEmpty()) {
+      LOG.warn("No search intervals generated for dataSource[%s]", dataSource);
+      return new ReindexingTimelineView(dataSource, referenceTime, null, Collections.emptyList(), null);
+    }
+
+    // Calculate effective end time based on skip offset
+    DateTime effectiveEndTime = referenceTime;
+    ReindexingTimelineView.SkipOffsetInfo skipOffsetInfo = null;
+
+    if (skipOffsetFromNow != null) {
+      effectiveEndTime = referenceTime.minus(skipOffsetFromNow);
+      skipOffsetInfo = new ReindexingTimelineView.SkipOffsetInfo(
+          "skipOffsetFromNow",
+          skipOffsetFromNow,
+          true,
+          effectiveEndTime,
+          null
+      );
+    } else if (skipOffsetFromLatest != null) {
+      // skipOffsetFromLatest requires actual timeline data, so we can't apply it in preview mode
+      skipOffsetInfo = new ReindexingTimelineView.SkipOffsetInfo(
+          "skipOffsetFromLatest",
+          skipOffsetFromLatest,
+          false,
+          null,
+          "Requires actual segment timeline data"
+      );
+    }
+
+    // Build configs for each interval
+    List<ReindexingTimelineView.IntervalConfig> intervalConfigs = new ArrayList<>();
+    for (IntervalPartitioningInfo intervalInfo : searchIntervals) {
+      Interval searchInterval = intervalInfo.getInterval();
+
+      // Check if interval extends past skip offset
+      if (intervalEndsAfter(searchInterval, effectiveEndTime)) {
+        // Include in timeline, but indicate skipped by applying no rules.
+        intervalConfigs.add(new ReindexingTimelineView.IntervalConfig(
+            searchInterval,
+            0,
+            null,
+            Collections.emptyList()
+        ));
+        continue;
+      }
+
+      InlineSchemaDataSourceCompactionConfig.Builder builder = createBaseBuilder();
+      ReindexingConfigBuilder configBuilder = new ReindexingConfigBuilder(
+          ruleProvider,
+          searchInterval,
+          referenceTime,
+          searchIntervals,
+          null

Review Comment:
   P2 Timeline preview drops template tuningConfig
   
   getReindexingTimelineView builds each interval config with a ReindexingConfigBuilder whose baseTuningConfig is null, while createCompactionJobs passes the template's tuningConfig into the same builder. Any cascading supervisor with static tuning settings such as maxRowsInMemory, splitHintSpec, maxNumConcurrentSubTasks, indexSpecForIntermediatePersists, or retry/chat settings will have those settings in the actual compaction jobs but not in the new reindexingTimeline API response or downloaded UI config. This makes the preview/API incompatible with the real effective config. Pass tuningConfig here as createCompactionJobs does, and add a test that a non-partitionsSpec tuning field survives in the timeline view.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org