You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "abhishekrb19 (via GitHub)" <gi...@apache.org> on 2024/02/22 14:12:00 UTC
[PR] WIP: Auto-kill bug fixes and improvements (druid)
abhishekrb19 opened a new pull request, #15941:
URL: https://github.com/apache/druid/pull/15941
This PR makes a few functional and non-functional changes in the auto-kill `KillUnusedSegments` duty.
Functional changes:
- Add a new metric `killTask/candidateSegments/count` that tracks the number of candidate unused segments per kill cycle. It's candidate because it's the kill task that ultimately kills the unused segments and the number can change.
- Move a debug log to info for improved visibility. This should be fine as it gets logged once per indexing/kill task period.
- Bug fix - auto-kill throws NPE when there are no datasources present and defaults mismatch.
- Change the kill duty DruidException validations from user to operator persona as they're configured by operators typically.
- Remove redundant checks in the code for code readability.
- Remove all the unnecessary `@VisibleForTesting` annotations as the test code now verifies the duty directly instead of mocks + various hooks into the code.
- Finalize some parameters and variables to clarify the intent.
Testing changes:
- Updated tests from using mocks (the mocks were difficult to use and buggy - for example, last updated timestamp was never used.
- Fix a bug where `TestDruidCoordinatorConfig.DEFAULT_COORDINATOR_KILL_BUFFER_PERIOD ` was incorrectly set to 1 day instead of 90 days. While at it, I changed all the test coordinator defaults to match the same ISO 8601 period format as `DruidCoordinatorConfig`, so it's easy to spot check.
- Add more test coverage for the different config parameters.
- Add a couple of unit tests that are ignored currently as they fail. The duty doesn't consider segments spanning eternity, or end in `DateTimes.MAX` (outside the normal datetime string comparison range).
TODO: need to clean up some test setup.
### Motivation
In general, I found it difficult to track the progress of the auto-kill coordinator duty. Also, see some discussion in the community slack: https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1707941058448649.
My preliminary conclusion from looking at a couple of small-to-medium clusters is that the auto-kill defaults are overly conservative and need to be revisited as https://github.com/apache/druid/issues/15912 notes. I would like to address some of the concerns raised in https://github.com/apache/druid/issues/15911 and https://github.com/apache/druid/issues/15912 separately once we have tests from this patch and some additional data in place.
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
<hr>
##### Key changed/added classes in this PR
* `KillUnusedSegments.java`
* `KillUnusedSegmentsTest.java`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1501754884
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +156,128 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
- Collection<String> dataSourcesToKill =
- params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
- CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
- killTaskSlotRatio,
- maxKillTaskSlots
- );
- int availableKillTaskSlots = getAvailableKillTaskSlots(
- killTaskCapacity,
- CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
- );
+ final CoordinatorDynamicConfig dynamicConfig = params.getCoordinatorDynamicConfig();
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(dynamicConfig, stats);
+ Collection<String> dataSourcesToKill = dynamicConfig.getSpecificDataSourcesToKillUnusedSegmentsIn();
- if (0 < availableKillTaskSlots) {
+ if (availableKillTaskSlots > 0) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
+ /**
+ * Spawn kill tasks for each datasource in {@code dataSourcesToKill} upto {@code availableKillTaskSlots}.
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
- )
- {
+ final Collection<String> remainingDatasourcesToKill = new ArrayList<>(dataSourcesToKill);
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ remainingDatasourcesToKill.remove(dataSource);
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
- submittedTasks,
- dataSourcesToKill.size(),
- availableKillTaskSlots < dataSourcesToKill.size()
- ? StringUtils.format(
- " Datasources skipped: %s",
- ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size())
- )
- : ""
- );
- }
+ log.info(
+ "Submitted [%d] kill tasks for [%d] datasources. Remaining datasources to kill: %s",
+ submittedTasks,
+ dataSourcesToKill.size(),
+ remainingDatasourcesToKill
+ );
- // report stats
- return submittedTasks;
+ stats.add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
}
/**
+ * <p>
* Calculates the interval for which segments are to be killed in a datasource.
+ * Because this function uses string comparisons for date time, it doesn't find unused segments that are outside
+ * the range [{@link DateTimes#COMPARE_DATE_AS_STRING_MIN}, {@link DateTimes#COMPARE_DATE_AS_STRING_MAX}),
+ * such as {@link org.apache.druid.java.util.common.granularity.Granularities#ALL} partitioned segments and segments that
+ * end in {@link DateTimes#MAX}.
+ *</p><p>
+ * For more information, see <a href="https://github.com/apache/druid/issues/15951"> Issue#15951</a>.
+ * </p>
*/
@Nullable
- private Interval findIntervalForKill(String dataSource, DateTime maxUsedStatusLastUpdatedTime)
+ private Interval findIntervalForKill(
+ final String dataSource,
+ final DateTime maxUsedStatusLastUpdatedTime,
+ final CoordinatorRunStats stats
+ )
{
final DateTime maxEndTime = ignoreDurationToRetain
? DateTimes.COMPARE_DATE_AS_STRING_MAX
: DateTimes.nowUtc().minus(durationToRetain);
- List<Interval> unusedSegmentIntervals = segmentsMetadataManager
- .getUnusedSegmentIntervals(dataSource, datasourceToLastKillIntervalEnd.get(dataSource), maxEndTime, maxSegmentsToKill, maxUsedStatusLastUpdatedTime);
+
+ final List<Interval> unusedSegmentIntervals = segmentsMetadataManager.getUnusedSegmentIntervals(
+ dataSource,
+ datasourceToLastKillIntervalEnd.get(dataSource),
+ maxEndTime,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ );
+
+ // Each unused segment interval returned here has a 1:1 correspondence with an unused segment. So we can assume
Review Comment:
Please see `getUnusedSegmentIntervals()` - it returns a list of unused segment intervals when it could actually just be the unused segments itself, but more lightweight computation. Also, added `testKillMultipleSegmentsInSameInterval()` to demonstrate the scenario.
I considered changing it to directly fetch the unused segments instead of the intervals, which would also allow us to compute any additional statistics like size, etc. If you think that's useful, I can do it in a follow-up patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1501755441
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -221,7 +242,29 @@ public List<Interval> getUnusedSegmentIntervals(
final DateTime maxUsedStatusLastUpdatedTime
)
{
- return null;
+ final List<DataSegmentPlus> sortedUnusedSegmentPluses = new ArrayList<>(unusedSegments.values());
+ sortedUnusedSegmentPluses.sort(
+ Comparator.comparingLong(
+ dataSegmentPlus -> dataSegmentPlus.getDataSegment().getInterval().getStartMillis()
+ )
+ );
+
+ final List<Interval> unusedSegmentIntervals = new ArrayList<>();
+
+ for (final DataSegmentPlus unusedSegmentPlus : sortedUnusedSegmentPluses) {
+ final DataSegment unusedSegment = unusedSegmentPlus.getDataSegment();
+ if (dataSource.equals(unusedSegment.getDataSource())) {
+ final Interval interval = unusedSegment.getInterval();
+
+ if ((minStartTime == null || interval.getStart().isAfter(minStartTime)) &&
+ interval.getEnd().isBefore(maxEndTime) &&
+ unusedSegmentPlus.getUsedStatusLastUpdatedDate().isBefore(maxUsedStatusLastUpdatedTime)) {
+ unusedSegmentIntervals.add(interval);
+ }
+ }
+ }
+ log.info("Found [%d] unused segment intervals: [%s]", unusedSegmentIntervals.size(), unusedSegmentIntervals);
Review Comment:
Removed it as the new metric should suffice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499400507
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
+ */
+ @Test
+ public void testRunsWithKillPeriod()
+ {
+ configBuilder.withCoordinatorKillPeriod(Duration.standardHours(1));
+
+ setupUnusedSegments();
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
+ );
+
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ secondRun.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
+ }
+
+ /**
+ * Similar to {@link #testMultipleRuns()}
+ */
+ @Test
+ public void testMultipleDatasources()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, NEXT_DAY, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 2, 10, ImmutableMap.of(DS1, 2L, DS2, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ validateAndResetState(DS2, new Interval(YEAR_OLD.getStart(), DAY_OLD.getEnd()));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 4, 20, ImmutableMap.of(DS1, 4L, DS2, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+ validateAndResetState(DS2, NEXT_DAY);
+
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 5, 30, ImmutableMap.of(DS1, 5L, DS2, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * Even though "more recent" segments are also considered as candidates in the wide kill interval here,
+ * the kill task will narrow down to clean up only the segments that strictly honor the max kill time.
+ * See {@code KillUnusedSegmentsTaskTest#testKillMultipleUnusedSegmentsWithNullMaxUsedStatusLastUpdatedTime}
+ * for example.
+ */
+ @Test
+ public void testSpreadOutLastMaxSegments()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillBufferPeriod(Duration.standardDays(3));
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, HOUR_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(10));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 4L)),
+ firstKill.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), NEXT_MONTH.getEnd()));
+ }
+
+ @Test
+ public void testAddOldVersionsLater()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+
+ log.info("Second run...");
+ // Add two old unused segments now. These only get killed much later on when the kill
+ // duty eventually round robins its way through until the latest time.
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 2, 30, ImmutableMap.of(DS1, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+
+ log.info("Fourth run...");
+
+ final DruidCoordinatorRuntimeParams fourthKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(40, 3, 40, ImmutableMap.of(DS1, 5L)),
+ fourthKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ }
+
+ @Test
+ public void testNoDatasources()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ @Test
+ public void testWhiteList()
+ {
+ // Only segments more than a day old are killed
+ dynamicConfigBuilder.withSpecificDataSourcesToKillUnusedSegmentsIn(Collections.singleton(DS2));
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 0L, DS2, 1L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS2, YEAR_OLD);
+ validateAndResetState(DS1, null);
}
@Test
public void testDurationToRetain()
{
// Only segments more than a day old are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- dayOldSegment.getInterval().getEnd()
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(), monthOldSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, dayOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testNegativeDurationToRetain()
{
- // Duration to retain = -1 day, reinit target for config to take effect
- Mockito.doReturn(DURATION_TO_RETAIN.negated())
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN.negated());
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 5L)),
+ runParams.getCoordinatorStats()
);
- // Segments upto 1 day in the future are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextDaySegment.getInterval().getEnd()
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextDaySegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextDaySegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testIgnoreDurationToRetain()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 6L)),
+ runParams.getCoordinatorStats()
);
- // All future and past unused segments are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
+ // All past and future unused segments should be killed
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextMonthSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testMaxSegmentsToKill()
{
- Mockito.doReturn(1)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillMaxSegments(1);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- // Only 1 unused segment is killed
- runAndVerifyKillInterval(yearOldSegment.getInterval());
- verifyState(ImmutableMap.of(DATASOURCE, yearOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
+ validateAndResetState(
+ DS1,
+ yearOldSegment.getInterval()
+ );
}
@Test
public void testMultipleRuns()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- Mockito.doReturn(2)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ log.info("First run...");
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- yearOldSegment.getInterval().getStart(),
- monthOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, monthOldSegment.getInterval().getEnd()));
+ log.info("Second run...");
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 4L)),
+ secondRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ dayOldSegment.getInterval().getStart(),
+ hourOldSegment.getInterval().getEnd()
+ )
+ );
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- dayOldSegment.getInterval().getStart(),
- hourOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, hourOldSegment.getInterval().getEnd()));
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdRun = killDuty.run(paramsBuilder.build());
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- nextDaySegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10, 3);
+ validateStats(
+ new ExpectedStats(30, 3, 30, ImmutableMap.of(DS1, 6L)),
+ thirdRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ nextDaySegment.getInterval().getStart(),
+ nextMonthSegment.getInterval().getEnd()
+ )
+ );
}
@Test
public void testKillTaskSlotRatioNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(0.10, 10, 1, 5);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 0);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.10);
+ dynamicConfigBuilder.withMaxKillTaskSlots(10);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(1, 5);
+
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
public void testMaxKillTaskSlotsNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(1.0, 3, 3, 10);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 3);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(3, 10);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 3, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
- public void testGetKillTaskCapacity()
+ public void testKillTaskSlotStats1()
{
- Assert.assertEquals(
- 10,
- KillUnusedSegments.getKillTaskCapacity(10, 1.0, Integer.MAX_VALUE)
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(Integer.MAX_VALUE);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 0,
- KillUnusedSegments.getKillTaskCapacity(10, 0.0, Integer.MAX_VALUE)
- );
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats());
+ }
- Assert.assertEquals(
- 10,
- KillUnusedSegments.getKillTaskCapacity(10, Double.POSITIVE_INFINITY, Integer.MAX_VALUE)
+ @Test
+ public void testKillTaskSlotStats2()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(Integer.MAX_VALUE);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 0,
- KillUnusedSegments.getKillTaskCapacity(10, 1.0, 0)
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
+ }
- Assert.assertEquals(
- 1,
- KillUnusedSegments.getKillTaskCapacity(10, 0.1, 3)
+ @Test
+ public void testKillTaskSlotStats3()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(0);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 2,
- KillUnusedSegments.getKillTaskCapacity(10, 0.3, 2)
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
}
- private void runAndVerifyKillInterval(Interval expectedKillInterval)
+ @Test
+ public void testKillTaskSlotStats4()
{
- int limit = config.getCoordinatorKillMaxSegments();
- Mockito.doReturn(Futures.immediateFuture("ok"))
- .when(overlordClient)
- .runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(Interval.class),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any(DateTime.class));
- target.runInternal(params);
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.1);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Mockito.verify(overlordClient, Mockito.times(1)).runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.eq(DATASOURCE),
- ArgumentMatchers.eq(expectedKillInterval),
- ArgumentMatchers.eq(limit),
- ArgumentMatchers.any()
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(1, 0, 1, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
}
- private void verifyStats(int availableSlots, int submittedTasks, int maxSlots)
+ @Test
+ public void testKillTaskSlotStats5()
{
- verifyStats(availableSlots, submittedTasks, maxSlots, 1);
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.3);
+ dynamicConfigBuilder.withMaxKillTaskSlots(2);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(2, 0, 2, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
}
- private void verifyStats(int availableSlots, int submittedTasks, int maxSlots, int times)
+ @Test
+ public void testKillFirstHalfEternitySegment()
{
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.AVAILABLE_SLOTS, availableSlots);
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.MAX_SLOTS, maxSlots);
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ final Interval firstHalfEternity = new Interval(DateTimes.MIN, DateTimes.of("2024"));
+ createAndAddUnusedSegment(DS1, firstHalfEternity, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of()),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, firstHalfEternity);
}
- private void verifyState(Map<String, DateTime> expectedDatasourceToLastKillIntervalEnd)
+ /**
+ * Regardless of ignoreDurationToRetain configuration, auto-kill doesn't delete unused eternity segments because the
+ * unused segment retrieval code uses {@link DateTimes#COMPARE_DATE_AS_STRING_MAX} as the datetime string comparison
+ * for the end endpoint when retrieving unused segment intervals.
+ * TODO: link GitHub issue
+ */
+ @Ignore
+ @Test
+ public void testKillEternitySegment()
{
- Assert.assertEquals(expectedDatasourceToLastKillIntervalEnd, target.getDatasourceToLastKillIntervalEnd());
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, Intervals.ETERNITY);
}
- private void runAndVerifyNoKill()
+ /**
+ * Similar to {@link #testKillEternitySegment()}
+ * <p>
+ * Regardless of ignoreDurationToRetain configuration, auto-kill doesn't delete unused segments with an interval end with
+ * {@link DateTimes#MAX}. This is because the kill duty uses {@link DateTimes#COMPARE_DATE_AS_STRING_MAX} as the
+ * datetime string comparison for the end endpoint when retrieving unused segment intervals.
+ * TODO: link GitHub issue
+ * </p>
+ */
+ @Ignore
+ @Test
+ public void testKillSecondHalfEternitySegments()
{
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never()).runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(Interval.class),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any(DateTime.class)
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ final Interval secondHalfEternity = new Interval(DateTimes.of("1970"), DateTimes.MAX);
+
+ createAndAddUnusedSegment(DS1, secondHalfEternity, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, secondHalfEternity);
}
- private void mockTaskSlotUsage(
- double killTaskSlotRatio,
- int maxKillTaskSlots,
- int numPendingCoordKillTasks,
- int maxWorkerCapacity
- )
+ private void validateStats(final ExpectedStats expectedStats, final CoordinatorRunStats actualRunStats)
+ {
+ Assert.assertEquals(expectedStats.availableSlots, actualRunStats.get(Stats.Kill.AVAILABLE_SLOTS));
+ Assert.assertEquals(expectedStats.submittedTasks, actualRunStats.get(Stats.Kill.SUBMITTED_TASKS));
+ Assert.assertEquals(expectedStats.maxSlots, actualRunStats.get(Stats.Kill.MAX_SLOTS));
+
+ for (final Map.Entry<String, Long> expectedEntry : expectedStats.dataSourceToCandidateSegments.entrySet()) {
+ Assert.assertEquals(
+ expectedEntry.getKey(),
+ expectedEntry.getValue().longValue(),
+ actualRunStats.get(
+ Stats.Kill.CANDIDATE_SEGMENTS_KILLED,
+ RowKey.of(Dimension.DATASOURCE, expectedEntry.getKey())
+ )
+ );
+ }
+ }
+
+ private void validateAndResetState(final String dataSource, @Nullable final Interval expectedKillInterval)
{
- Mockito.doReturn(killTaskSlotRatio)
- .when(coordinatorDynamicConfig).getKillTaskSlotRatio();
- Mockito.doReturn(maxKillTaskSlots)
- .when(coordinatorDynamicConfig).getMaxKillTaskSlots();
- Mockito.doReturn(Futures.immediateFuture(new IndexingTotalWorkerCapacityInfo(1, maxWorkerCapacity)))
- .when(overlordClient)
- .getTotalWorkerCapacity();
- List<TaskStatusPlus> runningCoordinatorIssuedKillTasks = new ArrayList<>();
- for (int i = 0; i < numPendingCoordKillTasks; i++) {
- runningCoordinatorIssuedKillTasks.add(new TaskStatusPlus(
- KillUnusedSegments.TASK_ID_PREFIX + "_taskId_" + i,
- "groupId_" + i,
- KillUnusedSegments.KILL_TASK_TYPE,
- DateTimes.EPOCH,
- DateTimes.EPOCH,
- TaskState.RUNNING,
- RunnerTaskState.RUNNING,
- -1L,
- TaskLocation.unknown(),
- "datasource",
- null
- ));
+ final Interval observedLastKillInterval = overlordClient.getLastKillInterval(dataSource);
+ final String observedLastKillTaskId = overlordClient.getLastKillTaskId(dataSource);
+
+ Assert.assertEquals(
+ expectedKillInterval,
+ observedLastKillInterval
+ );
+
+ String expectedKillTaskId = null;
+ if (expectedKillInterval != null) {
+ expectedKillTaskId = TestOverlordClient.getTaskId(
+ KillUnusedSegments.TASK_ID_PREFIX,
+ dataSource,
+ expectedKillInterval
+ );
}
- Mockito.doReturn(Futures.immediateFuture(
- CloseableIterators.withEmptyBaggage(runningCoordinatorIssuedKillTasks.iterator())))
- .when(overlordClient)
- .taskStatuses(null, null, 0);
+
+ Assert.assertEquals(
+ expectedKillTaskId,
+ observedLastKillTaskId
+ );
Review Comment:
Can be in a single line. Same comment in other places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on PR #15941:
URL: https://github.com/apache/druid/pull/15941#issuecomment-1965888104
Thanks, @kfaraz! I will add your rename suggestion in a follow-up patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1500197119
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
Review Comment:
Updated the test name. Also, I included java docs only for tests that aren't very straightforward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1503631992
##########
docs/operations/metrics.md:
##########
@@ -342,7 +342,7 @@ These metrics are for the Druid Coordinator and are reset each time the Coordina
|`killTask/availableSlot/count`| Number of available task slots that can be used for auto kill tasks in the auto kill run. This is the max number of task slots minus any currently running auto kill tasks. | |Varies|
|`killTask/maxSlot/count`| Maximum number of task slots available for auto kill tasks in the auto kill run. | |Varies|
|`kill/task/count`| Number of tasks issued in the auto kill run. | |Varies|
-|`kill/candidateUnusedSegments/count`|Number of candidate unused segments to be deleted from the metadata store in an auto kill run.|`dataSource`|Varies|
+|`kill/candidateUnusedSegments/count`|The number of candidate unused segments eligible for deletion from the metadata store during an auto kill run for a datasource.|`dataSource`|Varies|
Review Comment:
```suggestion
|`kill/eligibleUnusedSegments/count`|The number of unused segments of a datasource that are identified as eligible for deletion from the metadata store by the coordinator.|`dataSource`|Varies|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1500195745
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -34,33 +36,69 @@
import org.joda.time.Interval;
import javax.annotation.Nullable;
+import java.util.ArrayList;
import java.util.Collection;
+import java.util.Comparator;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
public class TestSegmentsMetadataManager implements SegmentsMetadataManager
{
+ private static final Logger log = new Logger(TestSegmentsMetadataManager.class);
+
+ /**
+ * segments = usedSegments + unusedSegments. Tracking unusedSegments separately for convenience and
+ * additional metadata.
+ */
private final ConcurrentMap<String, DataSegment> segments = new ConcurrentHashMap<>();
private final ConcurrentMap<String, DataSegment> usedSegments = new ConcurrentHashMap<>();
+ private final ConcurrentMap<String, DataSegmentPlus> unusedSegments = new ConcurrentHashMap<>();
private volatile DataSourcesSnapshot snapshot;
- public void addSegment(DataSegment segment)
+ /**
+ * Adds the used segment.
+ */
+ public void addSegment(DataSegment segment, boolean isUsed)
Review Comment:
Removed it, I was trying to consolidate some of the methods, but decided to just keep them separate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1500196043
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -282,41 +300,16 @@ private Interval findIntervalForKill(String dataSource, DateTime maxUsedStatusLa
private int getAvailableKillTaskSlots(int killTaskCapacity, int numActiveKillTasks)
{
- return Math.max(
- 0,
- killTaskCapacity - numActiveKillTasks
- );
+ return Math.max(0, killTaskCapacity - numActiveKillTasks);
}
private boolean canDutyRun()
{
return lastKillTime == null || !DateTimes.nowUtc().isBefore(lastKillTime.plus(period));
}
- @VisibleForTesting
- static int getKillTaskCapacity(int totalWorkerCapacity, double killTaskSlotRatio, int maxKillTaskSlots)
+ private static int getKillTaskCapacity(int totalWorkerCapacity, double killTaskSlotRatio, int maxKillTaskSlots)
Review Comment:
Yeah, good point. Removed `static` and also folded this function with the other helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499376973
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
Review Comment:
Just calling this out in case it got missed in the refactor.
The original segments had their _end_ a year ago from now, a month ago from now and so on.
The new intervals `YEAR_OLD`, `MONTH_OLD` actually have their end at now.
Please ensure that the tests have been updated to incorporate that.
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -44,50 +53,49 @@
import org.joda.time.Period;
import org.junit.Assert;
import org.junit.Before;
+import org.junit.Ignore;
import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.mockito.Answers;
-import org.mockito.ArgumentMatchers;
-import org.mockito.Mock;
-import org.mockito.Mockito;
-import org.mockito.junit.MockitoJUnitRunner;
+import javax.annotation.Nullable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.stream.Collectors;
+import java.util.Set;
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyInt;
-import static org.mockito.ArgumentMatchers.anyString;
/**
- *
+ * Unit tests to test the {@link KillUnusedSegments} duty.
*/
-@RunWith(MockitoJUnitRunner.class)
public class KillUnusedSegmentsTest
{
- private static final Duration INDEXING_PERIOD = Duration.standardSeconds(1);
- private static final Duration COORDINATOR_KILL_PERIOD = Duration.standardSeconds(1);
- private static final Duration DURATION_TO_RETAIN = Duration.standardDays(1);
+ private static final Logger log = new Logger(KillUnusedSegmentsTest.class);
+ private static final Duration INDEXING_PERIOD = Duration.standardSeconds(0);
+ private static final Duration COORDINATOR_KILL_PERIOD = Duration.standardSeconds(0);
+ private static final Duration DURATION_TO_RETAIN = Duration.standardHours(36);
+ private static final Duration BUFFER_PERIOD = Duration.standardSeconds(1);
private static final int MAX_SEGMENTS_TO_KILL = 10;
- private static final String DATASOURCE = "DS1";
-
- @Mock
- private SegmentsMetadataManager segmentsMetadataManager;
- @Mock
- private OverlordClient overlordClient;
- @Mock(answer = Answers.RETURNS_DEEP_STUBS)
- private DruidCoordinatorConfig config;
-
- @Mock
- private CoordinatorRunStats stats;
- @Mock
- private DruidCoordinatorRuntimeParams params;
- @Mock
- private CoordinatorDynamicConfig coordinatorDynamicConfig;
+ private static final String DS1 = "DS1";
+ private static final String DS2 = "DS2";
+
+ private TestSegmentsMetadataManager segmentsMetadataManager;
+
+ private TestOverlordClient overlordClient;
+
+ private TestDruidCoordinatorConfig.Builder configBuilder;
+
+ private DruidCoordinatorRuntimeParams.Builder paramsBuilder;
+
+ private final CoordinatorDynamicConfig.Builder dynamicConfigBuilder = CoordinatorDynamicConfig.builder();
+
+ private static final DateTime NOW = DateTimes.nowUtc();
+ private static final Interval YEAR_OLD = new Interval(Period.days(1), NOW.minusDays(365));
Review Comment:
More apt to call these `YEAR_TILL_DATE`, `MONTH_TILL_DATE` or something. `YEAR_OLD` is ambiguous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1501735595
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +156,128 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
- Collection<String> dataSourcesToKill =
- params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
- CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
- killTaskSlotRatio,
- maxKillTaskSlots
- );
- int availableKillTaskSlots = getAvailableKillTaskSlots(
- killTaskCapacity,
- CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
- );
+ final CoordinatorDynamicConfig dynamicConfig = params.getCoordinatorDynamicConfig();
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(dynamicConfig, stats);
+ Collection<String> dataSourcesToKill = dynamicConfig.getSpecificDataSourcesToKillUnusedSegmentsIn();
- if (0 < availableKillTaskSlots) {
+ if (availableKillTaskSlots > 0) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
+ /**
+ * Spawn kill tasks for each datasource in {@code dataSourcesToKill} upto {@code availableKillTaskSlots}.
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
- )
- {
+ final Collection<String> remainingDatasourcesToKill = new ArrayList<>(dataSourcesToKill);
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
Review Comment:
```suggestion
log.info(
"Submitted [%d] kill tasks and reached kill task slot limit [%d].",
submittedTasks, availableKillTaskSlots
);
```
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +156,128 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
- Collection<String> dataSourcesToKill =
- params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
- CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
- killTaskSlotRatio,
- maxKillTaskSlots
- );
- int availableKillTaskSlots = getAvailableKillTaskSlots(
- killTaskCapacity,
- CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
- );
+ final CoordinatorDynamicConfig dynamicConfig = params.getCoordinatorDynamicConfig();
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(dynamicConfig, stats);
+ Collection<String> dataSourcesToKill = dynamicConfig.getSpecificDataSourcesToKillUnusedSegmentsIn();
- if (0 < availableKillTaskSlots) {
+ if (availableKillTaskSlots > 0) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
+ /**
+ * Spawn kill tasks for each datasource in {@code dataSourcesToKill} upto {@code availableKillTaskSlots}.
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
- )
- {
+ final Collection<String> remainingDatasourcesToKill = new ArrayList<>(dataSourcesToKill);
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ remainingDatasourcesToKill.remove(dataSource);
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
- submittedTasks,
- dataSourcesToKill.size(),
- availableKillTaskSlots < dataSourcesToKill.size()
- ? StringUtils.format(
- " Datasources skipped: %s",
- ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size())
- )
- : ""
- );
- }
+ log.info(
+ "Submitted [%d] kill tasks for [%d] datasources. Remaining datasources to kill: %s",
+ submittedTasks,
+ dataSourcesToKill.size(),
+ remainingDatasourcesToKill
+ );
- // report stats
- return submittedTasks;
+ stats.add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
}
/**
+ * <p>
* Calculates the interval for which segments are to be killed in a datasource.
+ * Because this function uses string comparisons for date time, it doesn't find unused segments that are outside
+ * the range [{@link DateTimes#COMPARE_DATE_AS_STRING_MIN}, {@link DateTimes#COMPARE_DATE_AS_STRING_MAX}),
+ * such as {@link org.apache.druid.java.util.common.granularity.Granularities#ALL} partitioned segments and segments that
+ * end in {@link DateTimes#MAX}.
Review Comment:
```suggestion
* such as {@link org.apache.druid.java.util.common.granularity.Granularities#ALL} partitioned segments
* and segments that end in {@link DateTimes#MAX}.
```
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -87,29 +90,36 @@ public class KillUnusedSegments implements CoordinatorDuty
private final OverlordClient overlordClient;
public KillUnusedSegments(
- SegmentsMetadataManager segmentsMetadataManager,
- OverlordClient overlordClient,
- DruidCoordinatorConfig config
+ final SegmentsMetadataManager segmentsMetadataManager,
+ final OverlordClient overlordClient,
+ final DruidCoordinatorConfig config
)
{
if (config.getCoordinatorKillPeriod().getMillis() < config.getCoordinatorIndexingPeriod().getMillis()) {
- throw InvalidInput.exception(
- "druid.coordinator.kill.period[%s] must be >= druid.coordinator.period.indexingPeriod[%s]",
- config.getCoordinatorKillPeriod(),
- config.getCoordinatorIndexingPeriod()
- );
+ throw DruidException.forPersona(DruidException.Persona.OPERATOR)
+ .ofCategory(DruidException.Category.INVALID_INPUT)
+ .build(StringUtils.format(
Review Comment:
Nit: would look neater like this:
```suggestion
.build(
StringUtils.format(
```
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +156,128 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
- Collection<String> dataSourcesToKill =
- params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
- CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
- killTaskSlotRatio,
- maxKillTaskSlots
- );
- int availableKillTaskSlots = getAvailableKillTaskSlots(
- killTaskCapacity,
- CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
- );
+ final CoordinatorDynamicConfig dynamicConfig = params.getCoordinatorDynamicConfig();
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(dynamicConfig, stats);
+ Collection<String> dataSourcesToKill = dynamicConfig.getSpecificDataSourcesToKillUnusedSegmentsIn();
- if (0 < availableKillTaskSlots) {
+ if (availableKillTaskSlots > 0) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
+ /**
+ * Spawn kill tasks for each datasource in {@code dataSourcesToKill} upto {@code availableKillTaskSlots}.
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
- )
- {
+ final Collection<String> remainingDatasourcesToKill = new ArrayList<>(dataSourcesToKill);
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ remainingDatasourcesToKill.remove(dataSource);
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
- submittedTasks,
- dataSourcesToKill.size(),
- availableKillTaskSlots < dataSourcesToKill.size()
- ? StringUtils.format(
- " Datasources skipped: %s",
- ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size())
- )
- : ""
- );
- }
+ log.info(
+ "Submitted [%d] kill tasks for [%d] datasources. Remaining datasources to kill: %s",
+ submittedTasks,
+ dataSourcesToKill.size(),
+ remainingDatasourcesToKill
Review Comment:
Super nit: I generally prefer this style as the logging occupies fewer lines without compromising readability, but either way is fine really.
```suggestion
submittedTasks, dataSourcesToKill.size(), remainingDatasourcesToKill
```
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +156,128 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
- Collection<String> dataSourcesToKill =
- params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
- CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
- killTaskSlotRatio,
- maxKillTaskSlots
- );
- int availableKillTaskSlots = getAvailableKillTaskSlots(
- killTaskCapacity,
- CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
- );
+ final CoordinatorDynamicConfig dynamicConfig = params.getCoordinatorDynamicConfig();
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(dynamicConfig, stats);
+ Collection<String> dataSourcesToKill = dynamicConfig.getSpecificDataSourcesToKillUnusedSegmentsIn();
- if (0 < availableKillTaskSlots) {
+ if (availableKillTaskSlots > 0) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
+ /**
+ * Spawn kill tasks for each datasource in {@code dataSourcesToKill} upto {@code availableKillTaskSlots}.
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
- )
- {
+ final Collection<String> remainingDatasourcesToKill = new ArrayList<>(dataSourcesToKill);
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ remainingDatasourcesToKill.remove(dataSource);
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
- submittedTasks,
- dataSourcesToKill.size(),
- availableKillTaskSlots < dataSourcesToKill.size()
- ? StringUtils.format(
- " Datasources skipped: %s",
- ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size())
- )
- : ""
- );
- }
+ log.info(
+ "Submitted [%d] kill tasks for [%d] datasources. Remaining datasources to kill: %s",
+ submittedTasks,
+ dataSourcesToKill.size(),
+ remainingDatasourcesToKill
+ );
- // report stats
- return submittedTasks;
+ stats.add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
}
/**
+ * <p>
* Calculates the interval for which segments are to be killed in a datasource.
+ * Because this function uses string comparisons for date time, it doesn't find unused segments that are outside
Review Comment:
```suggestion
* Since this method compares datetime as strings, it cannot find unused segments that are outside
```
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -221,7 +242,29 @@ public List<Interval> getUnusedSegmentIntervals(
final DateTime maxUsedStatusLastUpdatedTime
)
{
- return null;
+ final List<DataSegmentPlus> sortedUnusedSegmentPluses = new ArrayList<>(unusedSegments.values());
+ sortedUnusedSegmentPluses.sort(
+ Comparator.comparingLong(
+ dataSegmentPlus -> dataSegmentPlus.getDataSegment().getInterval().getStartMillis()
+ )
+ );
+
+ final List<Interval> unusedSegmentIntervals = new ArrayList<>();
+
+ for (final DataSegmentPlus unusedSegmentPlus : sortedUnusedSegmentPluses) {
+ final DataSegment unusedSegment = unusedSegmentPlus.getDataSegment();
+ if (dataSource.equals(unusedSegment.getDataSource())) {
+ final Interval interval = unusedSegment.getInterval();
+
+ if ((minStartTime == null || interval.getStart().isAfter(minStartTime)) &&
+ interval.getEnd().isBefore(maxEndTime) &&
+ unusedSegmentPlus.getUsedStatusLastUpdatedDate().isBefore(maxUsedStatusLastUpdatedTime)) {
+ unusedSegmentIntervals.add(interval);
+ }
+ }
+ }
+ log.info("Found [%d] unused segment intervals: [%s]", unusedSegmentIntervals.size(), unusedSegmentIntervals);
Review Comment:
If this is really needed, maybe it should be a log line in `KillUnusedSegments` duty because `SqlSegmentsMetadataManager` doesn't seem to log this either.
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -124,8 +139,14 @@ public int markAsUnusedSegmentsInInterval(String dataSource, Interval interval)
public int markSegmentsAsUnused(Set<SegmentId> segmentIds)
{
int numModifiedSegments = 0;
+ final DateTime now = DateTimes.nowUtc();
+ final DateTime lastUpdatedDate = now.plus(10);
Review Comment:
Please add a comment for why plus 10?
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +156,128 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
- Collection<String> dataSourcesToKill =
- params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
- CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
- killTaskSlotRatio,
- maxKillTaskSlots
- );
- int availableKillTaskSlots = getAvailableKillTaskSlots(
- killTaskCapacity,
- CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
- );
+ final CoordinatorDynamicConfig dynamicConfig = params.getCoordinatorDynamicConfig();
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(dynamicConfig, stats);
+ Collection<String> dataSourcesToKill = dynamicConfig.getSpecificDataSourcesToKillUnusedSegmentsIn();
- if (0 < availableKillTaskSlots) {
+ if (availableKillTaskSlots > 0) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
+ /**
+ * Spawn kill tasks for each datasource in {@code dataSourcesToKill} upto {@code availableKillTaskSlots}.
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
- )
- {
+ final Collection<String> remainingDatasourcesToKill = new ArrayList<>(dataSourcesToKill);
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ remainingDatasourcesToKill.remove(dataSource);
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
- submittedTasks,
- dataSourcesToKill.size(),
- availableKillTaskSlots < dataSourcesToKill.size()
- ? StringUtils.format(
- " Datasources skipped: %s",
- ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size())
- )
- : ""
- );
- }
+ log.info(
+ "Submitted [%d] kill tasks for [%d] datasources. Remaining datasources to kill: %s",
+ submittedTasks,
+ dataSourcesToKill.size(),
+ remainingDatasourcesToKill
+ );
- // report stats
- return submittedTasks;
+ stats.add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
}
/**
+ * <p>
* Calculates the interval for which segments are to be killed in a datasource.
+ * Because this function uses string comparisons for date time, it doesn't find unused segments that are outside
+ * the range [{@link DateTimes#COMPARE_DATE_AS_STRING_MIN}, {@link DateTimes#COMPARE_DATE_AS_STRING_MAX}),
+ * such as {@link org.apache.druid.java.util.common.granularity.Granularities#ALL} partitioned segments and segments that
+ * end in {@link DateTimes#MAX}.
+ *</p><p>
+ * For more information, see <a href="https://github.com/apache/druid/issues/15951"> Issue#15951</a>.
+ * </p>
*/
@Nullable
- private Interval findIntervalForKill(String dataSource, DateTime maxUsedStatusLastUpdatedTime)
+ private Interval findIntervalForKill(
+ final String dataSource,
+ final DateTime maxUsedStatusLastUpdatedTime,
+ final CoordinatorRunStats stats
+ )
{
final DateTime maxEndTime = ignoreDurationToRetain
? DateTimes.COMPARE_DATE_AS_STRING_MAX
: DateTimes.nowUtc().minus(durationToRetain);
- List<Interval> unusedSegmentIntervals = segmentsMetadataManager
- .getUnusedSegmentIntervals(dataSource, datasourceToLastKillIntervalEnd.get(dataSource), maxEndTime, maxSegmentsToKill, maxUsedStatusLastUpdatedTime);
+
+ final List<Interval> unusedSegmentIntervals = segmentsMetadataManager.getUnusedSegmentIntervals(
+ dataSource,
+ datasourceToLastKillIntervalEnd.get(dataSource),
+ maxEndTime,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ );
+
+ // Each unused segment interval returned here has a 1:1 correspondence with an unused segment. So we can assume
Review Comment:
How do we know this? A single interval can have many killable unused segments.
##########
docs/operations/metrics.md:
##########
@@ -342,6 +342,7 @@ These metrics are for the Druid Coordinator and are reset each time the Coordina
|`killTask/availableSlot/count`| Number of available task slots that can be used for auto kill tasks in the auto kill run. This is the max number of task slots minus any currently running auto kill tasks. | |Varies|
|`killTask/maxSlot/count`| Maximum number of task slots available for auto kill tasks in the auto kill run. | |Varies|
|`kill/task/count`| Number of tasks issued in the auto kill run. | |Varies|
+|`kill/candidateUnusedSegments/count`|Number of candidate unused segments to be deleted from the metadata store in an auto kill run.|`dataSource`|Varies|
Review Comment:
Reading at this doc and the name of the metric, it's not very clear what it represents.
Is it the number of segments that the coordinator _decided to kill_ in a datasource or the number of segments that _should be killed_ in a datasource? Also, should it have `interval` as a dimension?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1501755337
##########
docs/operations/metrics.md:
##########
@@ -342,6 +342,7 @@ These metrics are for the Druid Coordinator and are reset each time the Coordina
|`killTask/availableSlot/count`| Number of available task slots that can be used for auto kill tasks in the auto kill run. This is the max number of task slots minus any currently running auto kill tasks. | |Varies|
|`killTask/maxSlot/count`| Maximum number of task slots available for auto kill tasks in the auto kill run. | |Varies|
|`kill/task/count`| Number of tasks issued in the auto kill run. | |Varies|
+|`kill/candidateUnusedSegments/count`|Number of candidate unused segments to be deleted from the metadata store in an auto kill run.|`dataSource`|Varies|
Review Comment:
> Is it the number of segments that the coordinator decided to kill in a datasource or the number of segments that should be killed in a datasource?
It's the latter - clarified the docs a wee bit. Let me know if you have a better suggestion. I called it candidate because it's ultimately the underlying kill task that can say this for sure.
>Also, should it have interval as a dimension?
No, there can be multiple intervals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499410216
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
+ */
+ @Test
+ public void testRunsWithKillPeriod()
+ {
+ configBuilder.withCoordinatorKillPeriod(Duration.standardHours(1));
+
+ setupUnusedSegments();
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
+ );
+
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ secondRun.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
+ }
+
+ /**
+ * Similar to {@link #testMultipleRuns()}
+ */
+ @Test
+ public void testMultipleDatasources()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, NEXT_DAY, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 2, 10, ImmutableMap.of(DS1, 2L, DS2, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ validateAndResetState(DS2, new Interval(YEAR_OLD.getStart(), DAY_OLD.getEnd()));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 4, 20, ImmutableMap.of(DS1, 4L, DS2, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+ validateAndResetState(DS2, NEXT_DAY);
+
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 5, 30, ImmutableMap.of(DS1, 5L, DS2, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * Even though "more recent" segments are also considered as candidates in the wide kill interval here,
+ * the kill task will narrow down to clean up only the segments that strictly honor the max kill time.
+ * See {@code KillUnusedSegmentsTaskTest#testKillMultipleUnusedSegmentsWithNullMaxUsedStatusLastUpdatedTime}
+ * for example.
+ */
+ @Test
+ public void testSpreadOutLastMaxSegments()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillBufferPeriod(Duration.standardDays(3));
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, HOUR_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(10));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 4L)),
+ firstKill.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), NEXT_MONTH.getEnd()));
+ }
+
+ @Test
+ public void testAddOldVersionsLater()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+
+ log.info("Second run...");
+ // Add two old unused segments now. These only get killed much later on when the kill
+ // duty eventually round robins its way through until the latest time.
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 2, 30, ImmutableMap.of(DS1, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+
+ log.info("Fourth run...");
+
+ final DruidCoordinatorRuntimeParams fourthKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(40, 3, 40, ImmutableMap.of(DS1, 5L)),
+ fourthKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ }
+
+ @Test
+ public void testNoDatasources()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ @Test
+ public void testWhiteList()
+ {
+ // Only segments more than a day old are killed
+ dynamicConfigBuilder.withSpecificDataSourcesToKillUnusedSegmentsIn(Collections.singleton(DS2));
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 0L, DS2, 1L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS2, YEAR_OLD);
+ validateAndResetState(DS1, null);
}
@Test
public void testDurationToRetain()
{
// Only segments more than a day old are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- dayOldSegment.getInterval().getEnd()
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(), monthOldSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, dayOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testNegativeDurationToRetain()
{
- // Duration to retain = -1 day, reinit target for config to take effect
- Mockito.doReturn(DURATION_TO_RETAIN.negated())
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN.negated());
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 5L)),
+ runParams.getCoordinatorStats()
);
- // Segments upto 1 day in the future are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextDaySegment.getInterval().getEnd()
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextDaySegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextDaySegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testIgnoreDurationToRetain()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 6L)),
+ runParams.getCoordinatorStats()
);
- // All future and past unused segments are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
+ // All past and future unused segments should be killed
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextMonthSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testMaxSegmentsToKill()
{
- Mockito.doReturn(1)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillMaxSegments(1);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- // Only 1 unused segment is killed
- runAndVerifyKillInterval(yearOldSegment.getInterval());
- verifyState(ImmutableMap.of(DATASOURCE, yearOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
+ validateAndResetState(
+ DS1,
+ yearOldSegment.getInterval()
+ );
}
@Test
public void testMultipleRuns()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- Mockito.doReturn(2)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ log.info("First run...");
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- yearOldSegment.getInterval().getStart(),
- monthOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, monthOldSegment.getInterval().getEnd()));
+ log.info("Second run...");
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 4L)),
+ secondRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ dayOldSegment.getInterval().getStart(),
+ hourOldSegment.getInterval().getEnd()
+ )
+ );
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- dayOldSegment.getInterval().getStart(),
- hourOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, hourOldSegment.getInterval().getEnd()));
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdRun = killDuty.run(paramsBuilder.build());
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- nextDaySegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10, 3);
+ validateStats(
+ new ExpectedStats(30, 3, 30, ImmutableMap.of(DS1, 6L)),
+ thirdRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ nextDaySegment.getInterval().getStart(),
+ nextMonthSegment.getInterval().getEnd()
+ )
+ );
}
@Test
public void testKillTaskSlotRatioNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(0.10, 10, 1, 5);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 0);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.10);
+ dynamicConfigBuilder.withMaxKillTaskSlots(10);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(1, 5);
+
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
public void testMaxKillTaskSlotsNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(1.0, 3, 3, 10);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 3);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(3, 10);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 3, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
- public void testGetKillTaskCapacity()
+ public void testKillTaskSlotStats1()
{
- Assert.assertEquals(
- 10,
- KillUnusedSegments.getKillTaskCapacity(10, 1.0, Integer.MAX_VALUE)
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(Integer.MAX_VALUE);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 0,
- KillUnusedSegments.getKillTaskCapacity(10, 0.0, Integer.MAX_VALUE)
- );
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats());
+ }
- Assert.assertEquals(
- 10,
- KillUnusedSegments.getKillTaskCapacity(10, Double.POSITIVE_INFINITY, Integer.MAX_VALUE)
+ @Test
+ public void testKillTaskSlotStats2()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(Integer.MAX_VALUE);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 0,
- KillUnusedSegments.getKillTaskCapacity(10, 1.0, 0)
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
+ }
- Assert.assertEquals(
- 1,
- KillUnusedSegments.getKillTaskCapacity(10, 0.1, 3)
+ @Test
+ public void testKillTaskSlotStats3()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(0);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 2,
- KillUnusedSegments.getKillTaskCapacity(10, 0.3, 2)
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
}
- private void runAndVerifyKillInterval(Interval expectedKillInterval)
+ @Test
+ public void testKillTaskSlotStats4()
{
- int limit = config.getCoordinatorKillMaxSegments();
- Mockito.doReturn(Futures.immediateFuture("ok"))
- .when(overlordClient)
- .runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(Interval.class),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any(DateTime.class));
- target.runInternal(params);
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.1);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Mockito.verify(overlordClient, Mockito.times(1)).runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.eq(DATASOURCE),
- ArgumentMatchers.eq(expectedKillInterval),
- ArgumentMatchers.eq(limit),
- ArgumentMatchers.any()
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(1, 0, 1, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
}
- private void verifyStats(int availableSlots, int submittedTasks, int maxSlots)
+ @Test
+ public void testKillTaskSlotStats5()
{
- verifyStats(availableSlots, submittedTasks, maxSlots, 1);
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.3);
+ dynamicConfigBuilder.withMaxKillTaskSlots(2);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(2, 0, 2, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
}
- private void verifyStats(int availableSlots, int submittedTasks, int maxSlots, int times)
+ @Test
+ public void testKillFirstHalfEternitySegment()
{
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.AVAILABLE_SLOTS, availableSlots);
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.MAX_SLOTS, maxSlots);
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ final Interval firstHalfEternity = new Interval(DateTimes.MIN, DateTimes.of("2024"));
+ createAndAddUnusedSegment(DS1, firstHalfEternity, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of()),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, firstHalfEternity);
}
- private void verifyState(Map<String, DateTime> expectedDatasourceToLastKillIntervalEnd)
+ /**
+ * Regardless of ignoreDurationToRetain configuration, auto-kill doesn't delete unused eternity segments because the
+ * unused segment retrieval code uses {@link DateTimes#COMPARE_DATE_AS_STRING_MAX} as the datetime string comparison
+ * for the end endpoint when retrieving unused segment intervals.
+ * TODO: link GitHub issue
+ */
+ @Ignore
+ @Test
+ public void testKillEternitySegment()
{
- Assert.assertEquals(expectedDatasourceToLastKillIntervalEnd, target.getDatasourceToLastKillIntervalEnd());
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, Intervals.ETERNITY);
}
- private void runAndVerifyNoKill()
+ /**
+ * Similar to {@link #testKillEternitySegment()}
+ * <p>
+ * Regardless of ignoreDurationToRetain configuration, auto-kill doesn't delete unused segments with an interval end with
+ * {@link DateTimes#MAX}. This is because the kill duty uses {@link DateTimes#COMPARE_DATE_AS_STRING_MAX} as the
+ * datetime string comparison for the end endpoint when retrieving unused segment intervals.
+ * TODO: link GitHub issue
+ * </p>
+ */
+ @Ignore
+ @Test
+ public void testKillSecondHalfEternitySegments()
{
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never()).runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(Interval.class),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any(DateTime.class)
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ final Interval secondHalfEternity = new Interval(DateTimes.of("1970"), DateTimes.MAX);
+
+ createAndAddUnusedSegment(DS1, secondHalfEternity, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, secondHalfEternity);
}
- private void mockTaskSlotUsage(
- double killTaskSlotRatio,
- int maxKillTaskSlots,
- int numPendingCoordKillTasks,
- int maxWorkerCapacity
- )
+ private void validateStats(final ExpectedStats expectedStats, final CoordinatorRunStats actualRunStats)
+ {
+ Assert.assertEquals(expectedStats.availableSlots, actualRunStats.get(Stats.Kill.AVAILABLE_SLOTS));
+ Assert.assertEquals(expectedStats.submittedTasks, actualRunStats.get(Stats.Kill.SUBMITTED_TASKS));
+ Assert.assertEquals(expectedStats.maxSlots, actualRunStats.get(Stats.Kill.MAX_SLOTS));
+
+ for (final Map.Entry<String, Long> expectedEntry : expectedStats.dataSourceToCandidateSegments.entrySet()) {
+ Assert.assertEquals(
+ expectedEntry.getKey(),
+ expectedEntry.getValue().longValue(),
+ actualRunStats.get(
+ Stats.Kill.CANDIDATE_SEGMENTS_KILLED,
+ RowKey.of(Dimension.DATASOURCE, expectedEntry.getKey())
+ )
+ );
+ }
+ }
+
+ private void validateAndResetState(final String dataSource, @Nullable final Interval expectedKillInterval)
Review Comment:
Please update the method name to indicate what is being validated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1500196710
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
+ */
+ @Test
+ public void testRunsWithKillPeriod()
+ {
+ configBuilder.withCoordinatorKillPeriod(Duration.standardHours(1));
+
+ setupUnusedSegments();
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
+ );
+
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ secondRun.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
+ }
+
+ /**
+ * Similar to {@link #testMultipleRuns()}
+ */
+ @Test
+ public void testMultipleDatasources()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, NEXT_DAY, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 2, 10, ImmutableMap.of(DS1, 2L, DS2, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ validateAndResetState(DS2, new Interval(YEAR_OLD.getStart(), DAY_OLD.getEnd()));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 4, 20, ImmutableMap.of(DS1, 4L, DS2, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+ validateAndResetState(DS2, NEXT_DAY);
+
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 5, 30, ImmutableMap.of(DS1, 5L, DS2, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * Even though "more recent" segments are also considered as candidates in the wide kill interval here,
+ * the kill task will narrow down to clean up only the segments that strictly honor the max kill time.
+ * See {@code KillUnusedSegmentsTaskTest#testKillMultipleUnusedSegmentsWithNullMaxUsedStatusLastUpdatedTime}
+ * for example.
+ */
+ @Test
+ public void testSpreadOutLastMaxSegments()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillBufferPeriod(Duration.standardDays(3));
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, HOUR_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(10));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 4L)),
+ firstKill.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), NEXT_MONTH.getEnd()));
+ }
+
+ @Test
+ public void testAddOldVersionsLater()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+
+ log.info("Second run...");
+ // Add two old unused segments now. These only get killed much later on when the kill
+ // duty eventually round robins its way through until the latest time.
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 2, 30, ImmutableMap.of(DS1, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+
+ log.info("Fourth run...");
+
+ final DruidCoordinatorRuntimeParams fourthKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(40, 3, 40, ImmutableMap.of(DS1, 5L)),
+ fourthKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ }
+
+ @Test
+ public void testNoDatasources()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ @Test
+ public void testWhiteList()
+ {
+ // Only segments more than a day old are killed
+ dynamicConfigBuilder.withSpecificDataSourcesToKillUnusedSegmentsIn(Collections.singleton(DS2));
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 0L, DS2, 1L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS2, YEAR_OLD);
+ validateAndResetState(DS1, null);
}
@Test
public void testDurationToRetain()
{
// Only segments more than a day old are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- dayOldSegment.getInterval().getEnd()
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(), monthOldSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, dayOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testNegativeDurationToRetain()
{
- // Duration to retain = -1 day, reinit target for config to take effect
- Mockito.doReturn(DURATION_TO_RETAIN.negated())
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN.negated());
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 5L)),
+ runParams.getCoordinatorStats()
);
- // Segments upto 1 day in the future are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextDaySegment.getInterval().getEnd()
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextDaySegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextDaySegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testIgnoreDurationToRetain()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 6L)),
+ runParams.getCoordinatorStats()
);
- // All future and past unused segments are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
+ // All past and future unused segments should be killed
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextMonthSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testMaxSegmentsToKill()
{
- Mockito.doReturn(1)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillMaxSegments(1);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- // Only 1 unused segment is killed
- runAndVerifyKillInterval(yearOldSegment.getInterval());
- verifyState(ImmutableMap.of(DATASOURCE, yearOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
+ validateAndResetState(
+ DS1,
+ yearOldSegment.getInterval()
+ );
}
@Test
public void testMultipleRuns()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- Mockito.doReturn(2)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ log.info("First run...");
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- yearOldSegment.getInterval().getStart(),
- monthOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, monthOldSegment.getInterval().getEnd()));
+ log.info("Second run...");
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 4L)),
+ secondRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ dayOldSegment.getInterval().getStart(),
+ hourOldSegment.getInterval().getEnd()
+ )
+ );
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- dayOldSegment.getInterval().getStart(),
- hourOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, hourOldSegment.getInterval().getEnd()));
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdRun = killDuty.run(paramsBuilder.build());
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- nextDaySegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10, 3);
+ validateStats(
+ new ExpectedStats(30, 3, 30, ImmutableMap.of(DS1, 6L)),
+ thirdRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ nextDaySegment.getInterval().getStart(),
+ nextMonthSegment.getInterval().getEnd()
+ )
+ );
}
@Test
public void testKillTaskSlotRatioNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(0.10, 10, 1, 5);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 0);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.10);
+ dynamicConfigBuilder.withMaxKillTaskSlots(10);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(1, 5);
+
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
public void testMaxKillTaskSlotsNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(1.0, 3, 3, 10);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 3);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(3, 10);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 3, ImmutableMap.of()),
Review Comment:
Yes, definitely, it reads more clearly now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499379888
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -44,50 +53,49 @@
import org.joda.time.Period;
import org.junit.Assert;
import org.junit.Before;
+import org.junit.Ignore;
import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.mockito.Answers;
-import org.mockito.ArgumentMatchers;
-import org.mockito.Mock;
-import org.mockito.Mockito;
-import org.mockito.junit.MockitoJUnitRunner;
+import javax.annotation.Nullable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.stream.Collectors;
+import java.util.Set;
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyInt;
-import static org.mockito.ArgumentMatchers.anyString;
/**
- *
+ * Unit tests to test the {@link KillUnusedSegments} duty.
*/
-@RunWith(MockitoJUnitRunner.class)
public class KillUnusedSegmentsTest
{
- private static final Duration INDEXING_PERIOD = Duration.standardSeconds(1);
- private static final Duration COORDINATOR_KILL_PERIOD = Duration.standardSeconds(1);
- private static final Duration DURATION_TO_RETAIN = Duration.standardDays(1);
+ private static final Logger log = new Logger(KillUnusedSegmentsTest.class);
+ private static final Duration INDEXING_PERIOD = Duration.standardSeconds(0);
+ private static final Duration COORDINATOR_KILL_PERIOD = Duration.standardSeconds(0);
+ private static final Duration DURATION_TO_RETAIN = Duration.standardHours(36);
+ private static final Duration BUFFER_PERIOD = Duration.standardSeconds(1);
private static final int MAX_SEGMENTS_TO_KILL = 10;
- private static final String DATASOURCE = "DS1";
-
- @Mock
- private SegmentsMetadataManager segmentsMetadataManager;
- @Mock
- private OverlordClient overlordClient;
- @Mock(answer = Answers.RETURNS_DEEP_STUBS)
- private DruidCoordinatorConfig config;
-
- @Mock
- private CoordinatorRunStats stats;
- @Mock
- private DruidCoordinatorRuntimeParams params;
- @Mock
- private CoordinatorDynamicConfig coordinatorDynamicConfig;
+ private static final String DS1 = "DS1";
+ private static final String DS2 = "DS2";
+
+ private TestSegmentsMetadataManager segmentsMetadataManager;
+
+ private TestOverlordClient overlordClient;
+
+ private TestDruidCoordinatorConfig.Builder configBuilder;
+
+ private DruidCoordinatorRuntimeParams.Builder paramsBuilder;
+
+ private final CoordinatorDynamicConfig.Builder dynamicConfigBuilder = CoordinatorDynamicConfig.builder();
+
+ private static final DateTime NOW = DateTimes.nowUtc();
+ private static final Interval YEAR_OLD = new Interval(Period.days(1), NOW.minusDays(365));
Review Comment:
More apt to call these `YEAR_TILL_DATE`, `MONTH_TILL_DATE` or something. `YEAR_OLD` is ambiguous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1500196365
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +154,140 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
Collection<String> dataSourcesToKill =
params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
+ final double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
+ final int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
+ final int killTaskCapacity = getKillTaskCapacity(
CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
killTaskSlotRatio,
maxKillTaskSlots
);
- int availableKillTaskSlots = getAvailableKillTaskSlots(
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(
killTaskCapacity,
CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
);
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
-
if (0 < availableKillTaskSlots) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
+
+ stats.add(Stats.Kill.AVAILABLE_SLOTS, availableKillTaskSlots);
+ stats.add(Stats.Kill.MAX_SLOTS, killTaskCapacity);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
- )
- {
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
-
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
+ /**
+ * Spawn kill tasks for each datasource to be killed upto {@code availableKillTaskSlots}.
+ * @param dataSourcesToKill finalized set of datasources to kill
+ * @param availableKillTaskSlots number of available kill task slots
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
+ log.info("Submitted [%d] kill tasks for [%d] datasources.%s",
submittedTasks,
dataSourcesToKill.size(),
availableKillTaskSlots < dataSourcesToKill.size()
? StringUtils.format(
" Datasources skipped: %s",
- ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size())
- )
+ ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size()))
: ""
- );
Review Comment:
Yeah, thanks, I wondered the same initially :) Updated per your suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1501755337
##########
docs/operations/metrics.md:
##########
@@ -342,6 +342,7 @@ These metrics are for the Druid Coordinator and are reset each time the Coordina
|`killTask/availableSlot/count`| Number of available task slots that can be used for auto kill tasks in the auto kill run. This is the max number of task slots minus any currently running auto kill tasks. | |Varies|
|`killTask/maxSlot/count`| Maximum number of task slots available for auto kill tasks in the auto kill run. | |Varies|
|`kill/task/count`| Number of tasks issued in the auto kill run. | |Varies|
+|`kill/candidateUnusedSegments/count`|Number of candidate unused segments to be deleted from the metadata store in an auto kill run.|`dataSource`|Varies|
Review Comment:
> Is it the number of segments that the coordinator decided to kill in a datasource or the number of segments that should be killed in a datasource?
It's the latter - clarified the docs a wee bit. Let me know if you have a better suggestion. I called it candidate because it's ultimately the underlying kill task that can say this for sure.
>Also, should it have interval as a dimension?
No, there can be multiple intervals, so an interval dimension can explode the metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1502267940
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +156,128 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
- Collection<String> dataSourcesToKill =
- params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
- CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
- killTaskSlotRatio,
- maxKillTaskSlots
- );
- int availableKillTaskSlots = getAvailableKillTaskSlots(
- killTaskCapacity,
- CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
- );
+ final CoordinatorDynamicConfig dynamicConfig = params.getCoordinatorDynamicConfig();
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(dynamicConfig, stats);
+ Collection<String> dataSourcesToKill = dynamicConfig.getSpecificDataSourcesToKillUnusedSegmentsIn();
- if (0 < availableKillTaskSlots) {
+ if (availableKillTaskSlots > 0) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
+ /**
+ * Spawn kill tasks for each datasource in {@code dataSourcesToKill} upto {@code availableKillTaskSlots}.
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
- )
- {
+ final Collection<String> remainingDatasourcesToKill = new ArrayList<>(dataSourcesToKill);
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ remainingDatasourcesToKill.remove(dataSource);
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
- submittedTasks,
- dataSourcesToKill.size(),
- availableKillTaskSlots < dataSourcesToKill.size()
- ? StringUtils.format(
- " Datasources skipped: %s",
- ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size())
- )
- : ""
- );
- }
+ log.info(
+ "Submitted [%d] kill tasks for [%d] datasources. Remaining datasources to kill: %s",
+ submittedTasks,
+ dataSourcesToKill.size(),
+ remainingDatasourcesToKill
+ );
- // report stats
- return submittedTasks;
+ stats.add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
}
/**
+ * <p>
* Calculates the interval for which segments are to be killed in a datasource.
+ * Because this function uses string comparisons for date time, it doesn't find unused segments that are outside
+ * the range [{@link DateTimes#COMPARE_DATE_AS_STRING_MIN}, {@link DateTimes#COMPARE_DATE_AS_STRING_MAX}),
+ * such as {@link org.apache.druid.java.util.common.granularity.Granularities#ALL} partitioned segments and segments that
+ * end in {@link DateTimes#MAX}.
+ *</p><p>
+ * For more information, see <a href="https://github.com/apache/druid/issues/15951"> Issue#15951</a>.
+ * </p>
*/
@Nullable
- private Interval findIntervalForKill(String dataSource, DateTime maxUsedStatusLastUpdatedTime)
+ private Interval findIntervalForKill(
+ final String dataSource,
+ final DateTime maxUsedStatusLastUpdatedTime,
+ final CoordinatorRunStats stats
+ )
{
final DateTime maxEndTime = ignoreDurationToRetain
? DateTimes.COMPARE_DATE_AS_STRING_MAX
: DateTimes.nowUtc().minus(durationToRetain);
- List<Interval> unusedSegmentIntervals = segmentsMetadataManager
- .getUnusedSegmentIntervals(dataSource, datasourceToLastKillIntervalEnd.get(dataSource), maxEndTime, maxSegmentsToKill, maxUsedStatusLastUpdatedTime);
+
+ final List<Interval> unusedSegmentIntervals = segmentsMetadataManager.getUnusedSegmentIntervals(
+ dataSource,
+ datasourceToLastKillIntervalEnd.get(dataSource),
+ maxEndTime,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ );
+
+ // Each unused segment interval returned here has a 1:1 correspondence with an unused segment. So we can assume
Review Comment:
Yeah, I think we can revisit that later. Let me clarify the current behavior in the method's javadocs to avoid any confusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1500196554
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
+ */
+ @Test
+ public void testRunsWithKillPeriod()
+ {
+ configBuilder.withCoordinatorKillPeriod(Duration.standardHours(1));
+
+ setupUnusedSegments();
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
+ );
+
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ secondRun.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
+ }
+
+ /**
+ * Similar to {@link #testMultipleRuns()}
+ */
+ @Test
+ public void testMultipleDatasources()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, NEXT_DAY, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 2, 10, ImmutableMap.of(DS1, 2L, DS2, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ validateAndResetState(DS2, new Interval(YEAR_OLD.getStart(), DAY_OLD.getEnd()));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 4, 20, ImmutableMap.of(DS1, 4L, DS2, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+ validateAndResetState(DS2, NEXT_DAY);
+
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 5, 30, ImmutableMap.of(DS1, 5L, DS2, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * Even though "more recent" segments are also considered as candidates in the wide kill interval here,
+ * the kill task will narrow down to clean up only the segments that strictly honor the max kill time.
+ * See {@code KillUnusedSegmentsTaskTest#testKillMultipleUnusedSegmentsWithNullMaxUsedStatusLastUpdatedTime}
+ * for example.
+ */
+ @Test
+ public void testSpreadOutLastMaxSegments()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillBufferPeriod(Duration.standardDays(3));
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, HOUR_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(10));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 4L)),
+ firstKill.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), NEXT_MONTH.getEnd()));
+ }
+
+ @Test
+ public void testAddOldVersionsLater()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+
+ log.info("Second run...");
+ // Add two old unused segments now. These only get killed much later on when the kill
+ // duty eventually round robins its way through until the latest time.
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 2, 30, ImmutableMap.of(DS1, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+
+ log.info("Fourth run...");
Review Comment:
Ack, removed the test logger altogether.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499336733
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +154,140 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
Review Comment:
Thanks for getting rid of these!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499400507
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
+ */
+ @Test
+ public void testRunsWithKillPeriod()
+ {
+ configBuilder.withCoordinatorKillPeriod(Duration.standardHours(1));
+
+ setupUnusedSegments();
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
+ );
+
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ secondRun.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
+ }
+
+ /**
+ * Similar to {@link #testMultipleRuns()}
+ */
+ @Test
+ public void testMultipleDatasources()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, NEXT_DAY, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 2, 10, ImmutableMap.of(DS1, 2L, DS2, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ validateAndResetState(DS2, new Interval(YEAR_OLD.getStart(), DAY_OLD.getEnd()));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 4, 20, ImmutableMap.of(DS1, 4L, DS2, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+ validateAndResetState(DS2, NEXT_DAY);
+
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 5, 30, ImmutableMap.of(DS1, 5L, DS2, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * Even though "more recent" segments are also considered as candidates in the wide kill interval here,
+ * the kill task will narrow down to clean up only the segments that strictly honor the max kill time.
+ * See {@code KillUnusedSegmentsTaskTest#testKillMultipleUnusedSegmentsWithNullMaxUsedStatusLastUpdatedTime}
+ * for example.
+ */
+ @Test
+ public void testSpreadOutLastMaxSegments()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillBufferPeriod(Duration.standardDays(3));
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, HOUR_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(10));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 4L)),
+ firstKill.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), NEXT_MONTH.getEnd()));
+ }
+
+ @Test
+ public void testAddOldVersionsLater()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+
+ log.info("Second run...");
+ // Add two old unused segments now. These only get killed much later on when the kill
+ // duty eventually round robins its way through until the latest time.
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 2, 30, ImmutableMap.of(DS1, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+
+ log.info("Fourth run...");
+
+ final DruidCoordinatorRuntimeParams fourthKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(40, 3, 40, ImmutableMap.of(DS1, 5L)),
+ fourthKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ }
+
+ @Test
+ public void testNoDatasources()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ @Test
+ public void testWhiteList()
+ {
+ // Only segments more than a day old are killed
+ dynamicConfigBuilder.withSpecificDataSourcesToKillUnusedSegmentsIn(Collections.singleton(DS2));
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 0L, DS2, 1L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS2, YEAR_OLD);
+ validateAndResetState(DS1, null);
}
@Test
public void testDurationToRetain()
{
// Only segments more than a day old are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- dayOldSegment.getInterval().getEnd()
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(), monthOldSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, dayOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testNegativeDurationToRetain()
{
- // Duration to retain = -1 day, reinit target for config to take effect
- Mockito.doReturn(DURATION_TO_RETAIN.negated())
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN.negated());
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 5L)),
+ runParams.getCoordinatorStats()
);
- // Segments upto 1 day in the future are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextDaySegment.getInterval().getEnd()
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextDaySegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextDaySegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testIgnoreDurationToRetain()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 6L)),
+ runParams.getCoordinatorStats()
);
- // All future and past unused segments are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
+ // All past and future unused segments should be killed
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextMonthSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testMaxSegmentsToKill()
{
- Mockito.doReturn(1)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillMaxSegments(1);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- // Only 1 unused segment is killed
- runAndVerifyKillInterval(yearOldSegment.getInterval());
- verifyState(ImmutableMap.of(DATASOURCE, yearOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
+ validateAndResetState(
+ DS1,
+ yearOldSegment.getInterval()
+ );
}
@Test
public void testMultipleRuns()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- Mockito.doReturn(2)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ log.info("First run...");
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- yearOldSegment.getInterval().getStart(),
- monthOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, monthOldSegment.getInterval().getEnd()));
+ log.info("Second run...");
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 4L)),
+ secondRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ dayOldSegment.getInterval().getStart(),
+ hourOldSegment.getInterval().getEnd()
+ )
+ );
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- dayOldSegment.getInterval().getStart(),
- hourOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, hourOldSegment.getInterval().getEnd()));
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdRun = killDuty.run(paramsBuilder.build());
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- nextDaySegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10, 3);
+ validateStats(
+ new ExpectedStats(30, 3, 30, ImmutableMap.of(DS1, 6L)),
+ thirdRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ nextDaySegment.getInterval().getStart(),
+ nextMonthSegment.getInterval().getEnd()
+ )
+ );
}
@Test
public void testKillTaskSlotRatioNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(0.10, 10, 1, 5);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 0);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.10);
+ dynamicConfigBuilder.withMaxKillTaskSlots(10);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(1, 5);
+
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
public void testMaxKillTaskSlotsNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(1.0, 3, 3, 10);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 3);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(3, 10);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 3, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
- public void testGetKillTaskCapacity()
+ public void testKillTaskSlotStats1()
{
- Assert.assertEquals(
- 10,
- KillUnusedSegments.getKillTaskCapacity(10, 1.0, Integer.MAX_VALUE)
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(Integer.MAX_VALUE);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 0,
- KillUnusedSegments.getKillTaskCapacity(10, 0.0, Integer.MAX_VALUE)
- );
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats());
+ }
- Assert.assertEquals(
- 10,
- KillUnusedSegments.getKillTaskCapacity(10, Double.POSITIVE_INFINITY, Integer.MAX_VALUE)
+ @Test
+ public void testKillTaskSlotStats2()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(Integer.MAX_VALUE);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 0,
- KillUnusedSegments.getKillTaskCapacity(10, 1.0, 0)
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
+ }
- Assert.assertEquals(
- 1,
- KillUnusedSegments.getKillTaskCapacity(10, 0.1, 3)
+ @Test
+ public void testKillTaskSlotStats3()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(0);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 2,
- KillUnusedSegments.getKillTaskCapacity(10, 0.3, 2)
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
}
- private void runAndVerifyKillInterval(Interval expectedKillInterval)
+ @Test
+ public void testKillTaskSlotStats4()
{
- int limit = config.getCoordinatorKillMaxSegments();
- Mockito.doReturn(Futures.immediateFuture("ok"))
- .when(overlordClient)
- .runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(Interval.class),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any(DateTime.class));
- target.runInternal(params);
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.1);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Mockito.verify(overlordClient, Mockito.times(1)).runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.eq(DATASOURCE),
- ArgumentMatchers.eq(expectedKillInterval),
- ArgumentMatchers.eq(limit),
- ArgumentMatchers.any()
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(1, 0, 1, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
}
- private void verifyStats(int availableSlots, int submittedTasks, int maxSlots)
+ @Test
+ public void testKillTaskSlotStats5()
{
- verifyStats(availableSlots, submittedTasks, maxSlots, 1);
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.3);
+ dynamicConfigBuilder.withMaxKillTaskSlots(2);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(2, 0, 2, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
}
- private void verifyStats(int availableSlots, int submittedTasks, int maxSlots, int times)
+ @Test
+ public void testKillFirstHalfEternitySegment()
{
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.AVAILABLE_SLOTS, availableSlots);
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.MAX_SLOTS, maxSlots);
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ final Interval firstHalfEternity = new Interval(DateTimes.MIN, DateTimes.of("2024"));
+ createAndAddUnusedSegment(DS1, firstHalfEternity, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of()),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, firstHalfEternity);
}
- private void verifyState(Map<String, DateTime> expectedDatasourceToLastKillIntervalEnd)
+ /**
+ * Regardless of ignoreDurationToRetain configuration, auto-kill doesn't delete unused eternity segments because the
+ * unused segment retrieval code uses {@link DateTimes#COMPARE_DATE_AS_STRING_MAX} as the datetime string comparison
+ * for the end endpoint when retrieving unused segment intervals.
+ * TODO: link GitHub issue
+ */
+ @Ignore
+ @Test
+ public void testKillEternitySegment()
{
- Assert.assertEquals(expectedDatasourceToLastKillIntervalEnd, target.getDatasourceToLastKillIntervalEnd());
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, Intervals.ETERNITY);
}
- private void runAndVerifyNoKill()
+ /**
+ * Similar to {@link #testKillEternitySegment()}
+ * <p>
+ * Regardless of ignoreDurationToRetain configuration, auto-kill doesn't delete unused segments with an interval end with
+ * {@link DateTimes#MAX}. This is because the kill duty uses {@link DateTimes#COMPARE_DATE_AS_STRING_MAX} as the
+ * datetime string comparison for the end endpoint when retrieving unused segment intervals.
+ * TODO: link GitHub issue
+ * </p>
+ */
+ @Ignore
+ @Test
+ public void testKillSecondHalfEternitySegments()
{
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never()).runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(Interval.class),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any(DateTime.class)
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ final Interval secondHalfEternity = new Interval(DateTimes.of("1970"), DateTimes.MAX);
+
+ createAndAddUnusedSegment(DS1, secondHalfEternity, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, secondHalfEternity);
}
- private void mockTaskSlotUsage(
- double killTaskSlotRatio,
- int maxKillTaskSlots,
- int numPendingCoordKillTasks,
- int maxWorkerCapacity
- )
+ private void validateStats(final ExpectedStats expectedStats, final CoordinatorRunStats actualRunStats)
+ {
+ Assert.assertEquals(expectedStats.availableSlots, actualRunStats.get(Stats.Kill.AVAILABLE_SLOTS));
+ Assert.assertEquals(expectedStats.submittedTasks, actualRunStats.get(Stats.Kill.SUBMITTED_TASKS));
+ Assert.assertEquals(expectedStats.maxSlots, actualRunStats.get(Stats.Kill.MAX_SLOTS));
+
+ for (final Map.Entry<String, Long> expectedEntry : expectedStats.dataSourceToCandidateSegments.entrySet()) {
+ Assert.assertEquals(
+ expectedEntry.getKey(),
+ expectedEntry.getValue().longValue(),
+ actualRunStats.get(
+ Stats.Kill.CANDIDATE_SEGMENTS_KILLED,
+ RowKey.of(Dimension.DATASOURCE, expectedEntry.getKey())
+ )
+ );
+ }
+ }
+
+ private void validateAndResetState(final String dataSource, @Nullable final Interval expectedKillInterval)
{
- Mockito.doReturn(killTaskSlotRatio)
- .when(coordinatorDynamicConfig).getKillTaskSlotRatio();
- Mockito.doReturn(maxKillTaskSlots)
- .when(coordinatorDynamicConfig).getMaxKillTaskSlots();
- Mockito.doReturn(Futures.immediateFuture(new IndexingTotalWorkerCapacityInfo(1, maxWorkerCapacity)))
- .when(overlordClient)
- .getTotalWorkerCapacity();
- List<TaskStatusPlus> runningCoordinatorIssuedKillTasks = new ArrayList<>();
- for (int i = 0; i < numPendingCoordKillTasks; i++) {
- runningCoordinatorIssuedKillTasks.add(new TaskStatusPlus(
- KillUnusedSegments.TASK_ID_PREFIX + "_taskId_" + i,
- "groupId_" + i,
- KillUnusedSegments.KILL_TASK_TYPE,
- DateTimes.EPOCH,
- DateTimes.EPOCH,
- TaskState.RUNNING,
- RunnerTaskState.RUNNING,
- -1L,
- TaskLocation.unknown(),
- "datasource",
- null
- ));
+ final Interval observedLastKillInterval = overlordClient.getLastKillInterval(dataSource);
+ final String observedLastKillTaskId = overlordClient.getLastKillTaskId(dataSource);
+
+ Assert.assertEquals(
+ expectedKillInterval,
+ observedLastKillInterval
+ );
+
+ String expectedKillTaskId = null;
+ if (expectedKillInterval != null) {
+ expectedKillTaskId = TestOverlordClient.getTaskId(
+ KillUnusedSegments.TASK_ID_PREFIX,
+ dataSource,
+ expectedKillInterval
+ );
}
- Mockito.doReturn(Futures.immediateFuture(
- CloseableIterators.withEmptyBaggage(runningCoordinatorIssuedKillTasks.iterator())))
- .when(overlordClient)
- .taskStatuses(null, null, 0);
+
+ Assert.assertEquals(
+ expectedKillTaskId,
+ observedLastKillTaskId
+ );
Review Comment:
Can be in a single line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499398893
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
Review Comment:
Maybe try to include this info in the test name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1502105293
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +156,128 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
- Collection<String> dataSourcesToKill =
- params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
- CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
- killTaskSlotRatio,
- maxKillTaskSlots
- );
- int availableKillTaskSlots = getAvailableKillTaskSlots(
- killTaskCapacity,
- CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
- );
+ final CoordinatorDynamicConfig dynamicConfig = params.getCoordinatorDynamicConfig();
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(dynamicConfig, stats);
+ Collection<String> dataSourcesToKill = dynamicConfig.getSpecificDataSourcesToKillUnusedSegmentsIn();
- if (0 < availableKillTaskSlots) {
+ if (availableKillTaskSlots > 0) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
+ /**
+ * Spawn kill tasks for each datasource in {@code dataSourcesToKill} upto {@code availableKillTaskSlots}.
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
- )
- {
+ final Collection<String> remainingDatasourcesToKill = new ArrayList<>(dataSourcesToKill);
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ remainingDatasourcesToKill.remove(dataSource);
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
- submittedTasks,
- dataSourcesToKill.size(),
- availableKillTaskSlots < dataSourcesToKill.size()
- ? StringUtils.format(
- " Datasources skipped: %s",
- ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size())
- )
- : ""
- );
- }
+ log.info(
+ "Submitted [%d] kill tasks for [%d] datasources. Remaining datasources to kill: %s",
+ submittedTasks,
+ dataSourcesToKill.size(),
+ remainingDatasourcesToKill
+ );
- // report stats
- return submittedTasks;
+ stats.add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
}
/**
+ * <p>
* Calculates the interval for which segments are to be killed in a datasource.
+ * Because this function uses string comparisons for date time, it doesn't find unused segments that are outside
+ * the range [{@link DateTimes#COMPARE_DATE_AS_STRING_MIN}, {@link DateTimes#COMPARE_DATE_AS_STRING_MAX}),
+ * such as {@link org.apache.druid.java.util.common.granularity.Granularities#ALL} partitioned segments and segments that
+ * end in {@link DateTimes#MAX}.
+ *</p><p>
+ * For more information, see <a href="https://github.com/apache/druid/issues/15951"> Issue#15951</a>.
+ * </p>
*/
@Nullable
- private Interval findIntervalForKill(String dataSource, DateTime maxUsedStatusLastUpdatedTime)
+ private Interval findIntervalForKill(
+ final String dataSource,
+ final DateTime maxUsedStatusLastUpdatedTime,
+ final CoordinatorRunStats stats
+ )
{
final DateTime maxEndTime = ignoreDurationToRetain
? DateTimes.COMPARE_DATE_AS_STRING_MAX
: DateTimes.nowUtc().minus(durationToRetain);
- List<Interval> unusedSegmentIntervals = segmentsMetadataManager
- .getUnusedSegmentIntervals(dataSource, datasourceToLastKillIntervalEnd.get(dataSource), maxEndTime, maxSegmentsToKill, maxUsedStatusLastUpdatedTime);
+
+ final List<Interval> unusedSegmentIntervals = segmentsMetadataManager.getUnusedSegmentIntervals(
+ dataSource,
+ datasourceToLastKillIntervalEnd.get(dataSource),
+ maxEndTime,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ );
+
+ // Each unused segment interval returned here has a 1:1 correspondence with an unused segment. So we can assume
Review Comment:
I see, thanks for the clarification! Looking at the implementation `SqlSegmentsMetadataManager`, that does indeed seem to be the case. I guess the javadoc in `SegmentsMetadataManager` didn't really call it out and I just assumed that it would be a list of unique intervals.
Although, it is a little weird for this API to return duplicate intervals, maybe something we should revisit? Or atleast call out in the `SegmentsMetadataManager` javadocs for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1501755739
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -124,8 +139,14 @@ public int markAsUnusedSegmentsInInterval(String dataSource, Interval interval)
public int markSegmentsAsUnused(Set<SegmentId> segmentIds)
{
int numModifiedSegments = 0;
+ final DateTime now = DateTimes.nowUtc();
+ final DateTime lastUpdatedDate = now.plus(10);
Review Comment:
Removed the magic number :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on PR #15941:
URL: https://github.com/apache/druid/pull/15941#issuecomment-1960738053
Thanks for the review, @kfaraz! I've addressed your comments and kept the new metric along with other minor functional changes in this patch. But let me know if you'd prefer to keep the changes separate. Re the NPE fix, my bad, it was from existing test code --`TestSegmentsMetadataManager` was returning null instead of an empty set for `retrieveAllDataSourceNames()`. Removed that from the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1502101870
##########
docs/operations/metrics.md:
##########
@@ -342,6 +342,7 @@ These metrics are for the Druid Coordinator and are reset each time the Coordina
|`killTask/availableSlot/count`| Number of available task slots that can be used for auto kill tasks in the auto kill run. This is the max number of task slots minus any currently running auto kill tasks. | |Varies|
|`killTask/maxSlot/count`| Maximum number of task slots available for auto kill tasks in the auto kill run. | |Varies|
|`kill/task/count`| Number of tasks issued in the auto kill run. | |Varies|
+|`kill/candidateUnusedSegments/count`|Number of candidate unused segments to be deleted from the metadata store in an auto kill run.|`dataSource`|Varies|
Review Comment:
> No, there can be multiple intervals, so an interval dimension can explode the metric.
Makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Refactor and add tests and metric to KillUnusedSegments duty (auto-kill) (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 merged PR #15941:
URL: https://github.com/apache/druid/pull/15941
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499408404
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
+ */
+ @Test
+ public void testRunsWithKillPeriod()
+ {
+ configBuilder.withCoordinatorKillPeriod(Duration.standardHours(1));
+
+ setupUnusedSegments();
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
+ );
+
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ secondRun.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
+ }
+
+ /**
+ * Similar to {@link #testMultipleRuns()}
+ */
+ @Test
+ public void testMultipleDatasources()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, NEXT_DAY, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 2, 10, ImmutableMap.of(DS1, 2L, DS2, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ validateAndResetState(DS2, new Interval(YEAR_OLD.getStart(), DAY_OLD.getEnd()));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 4, 20, ImmutableMap.of(DS1, 4L, DS2, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+ validateAndResetState(DS2, NEXT_DAY);
+
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 5, 30, ImmutableMap.of(DS1, 5L, DS2, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * Even though "more recent" segments are also considered as candidates in the wide kill interval here,
+ * the kill task will narrow down to clean up only the segments that strictly honor the max kill time.
+ * See {@code KillUnusedSegmentsTaskTest#testKillMultipleUnusedSegmentsWithNullMaxUsedStatusLastUpdatedTime}
+ * for example.
+ */
+ @Test
+ public void testSpreadOutLastMaxSegments()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillBufferPeriod(Duration.standardDays(3));
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, HOUR_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(10));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 4L)),
+ firstKill.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), NEXT_MONTH.getEnd()));
+ }
+
+ @Test
+ public void testAddOldVersionsLater()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+
+ log.info("Second run...");
+ // Add two old unused segments now. These only get killed much later on when the kill
+ // duty eventually round robins its way through until the latest time.
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 2, 30, ImmutableMap.of(DS1, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+
+ log.info("Fourth run...");
+
+ final DruidCoordinatorRuntimeParams fourthKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(40, 3, 40, ImmutableMap.of(DS1, 5L)),
+ fourthKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ }
+
+ @Test
+ public void testNoDatasources()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ @Test
+ public void testWhiteList()
+ {
+ // Only segments more than a day old are killed
+ dynamicConfigBuilder.withSpecificDataSourcesToKillUnusedSegmentsIn(Collections.singleton(DS2));
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 0L, DS2, 1L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS2, YEAR_OLD);
+ validateAndResetState(DS1, null);
}
@Test
public void testDurationToRetain()
{
// Only segments more than a day old are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- dayOldSegment.getInterval().getEnd()
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(), monthOldSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, dayOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testNegativeDurationToRetain()
{
- // Duration to retain = -1 day, reinit target for config to take effect
- Mockito.doReturn(DURATION_TO_RETAIN.negated())
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN.negated());
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 5L)),
+ runParams.getCoordinatorStats()
);
- // Segments upto 1 day in the future are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextDaySegment.getInterval().getEnd()
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextDaySegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextDaySegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testIgnoreDurationToRetain()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 6L)),
+ runParams.getCoordinatorStats()
);
- // All future and past unused segments are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
+ // All past and future unused segments should be killed
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextMonthSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testMaxSegmentsToKill()
{
- Mockito.doReturn(1)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillMaxSegments(1);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- // Only 1 unused segment is killed
- runAndVerifyKillInterval(yearOldSegment.getInterval());
- verifyState(ImmutableMap.of(DATASOURCE, yearOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
+ validateAndResetState(
+ DS1,
+ yearOldSegment.getInterval()
+ );
}
@Test
public void testMultipleRuns()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- Mockito.doReturn(2)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ log.info("First run...");
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- yearOldSegment.getInterval().getStart(),
- monthOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, monthOldSegment.getInterval().getEnd()));
+ log.info("Second run...");
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 4L)),
+ secondRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ dayOldSegment.getInterval().getStart(),
+ hourOldSegment.getInterval().getEnd()
+ )
+ );
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- dayOldSegment.getInterval().getStart(),
- hourOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, hourOldSegment.getInterval().getEnd()));
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdRun = killDuty.run(paramsBuilder.build());
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- nextDaySegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10, 3);
+ validateStats(
+ new ExpectedStats(30, 3, 30, ImmutableMap.of(DS1, 6L)),
+ thirdRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ nextDaySegment.getInterval().getStart(),
+ nextMonthSegment.getInterval().getEnd()
+ )
+ );
}
@Test
public void testKillTaskSlotRatioNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(0.10, 10, 1, 5);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 0);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.10);
+ dynamicConfigBuilder.withMaxKillTaskSlots(10);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(1, 5);
+
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
public void testMaxKillTaskSlotsNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(1.0, 3, 3, 10);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 3);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(3, 10);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 3, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
- public void testGetKillTaskCapacity()
+ public void testKillTaskSlotStats1()
{
- Assert.assertEquals(
- 10,
- KillUnusedSegments.getKillTaskCapacity(10, 1.0, Integer.MAX_VALUE)
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(Integer.MAX_VALUE);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 0,
- KillUnusedSegments.getKillTaskCapacity(10, 0.0, Integer.MAX_VALUE)
- );
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats());
+ }
- Assert.assertEquals(
- 10,
- KillUnusedSegments.getKillTaskCapacity(10, Double.POSITIVE_INFINITY, Integer.MAX_VALUE)
+ @Test
+ public void testKillTaskSlotStats2()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(Integer.MAX_VALUE);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 0,
- KillUnusedSegments.getKillTaskCapacity(10, 1.0, 0)
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
+ }
- Assert.assertEquals(
- 1,
- KillUnusedSegments.getKillTaskCapacity(10, 0.1, 3)
+ @Test
+ public void testKillTaskSlotStats3()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(0);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Assert.assertEquals(
- 2,
- KillUnusedSegments.getKillTaskCapacity(10, 0.3, 2)
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
}
- private void runAndVerifyKillInterval(Interval expectedKillInterval)
+ @Test
+ public void testKillTaskSlotStats4()
{
- int limit = config.getCoordinatorKillMaxSegments();
- Mockito.doReturn(Futures.immediateFuture("ok"))
- .when(overlordClient)
- .runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(Interval.class),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any(DateTime.class));
- target.runInternal(params);
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.1);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
- Mockito.verify(overlordClient, Mockito.times(1)).runKillTask(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.eq(DATASOURCE),
- ArgumentMatchers.eq(expectedKillInterval),
- ArgumentMatchers.eq(limit),
- ArgumentMatchers.any()
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(1, 0, 1, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
}
- private void verifyStats(int availableSlots, int submittedTasks, int maxSlots)
+ @Test
+ public void testKillTaskSlotStats5()
{
- verifyStats(availableSlots, submittedTasks, maxSlots, 1);
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.3);
+ dynamicConfigBuilder.withMaxKillTaskSlots(2);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(2, 0, 2, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
}
- private void verifyStats(int availableSlots, int submittedTasks, int maxSlots, int times)
+ @Test
+ public void testKillFirstHalfEternitySegment()
{
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.AVAILABLE_SLOTS, availableSlots);
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.SUBMITTED_TASKS, submittedTasks);
- Mockito.verify(stats, Mockito.times(times)).add(Stats.Kill.MAX_SLOTS, maxSlots);
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ final Interval firstHalfEternity = new Interval(DateTimes.MIN, DateTimes.of("2024"));
+ createAndAddUnusedSegment(DS1, firstHalfEternity, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of()),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, firstHalfEternity);
}
- private void verifyState(Map<String, DateTime> expectedDatasourceToLastKillIntervalEnd)
+ /**
+ * Regardless of ignoreDurationToRetain configuration, auto-kill doesn't delete unused eternity segments because the
+ * unused segment retrieval code uses {@link DateTimes#COMPARE_DATE_AS_STRING_MAX} as the datetime string comparison
+ * for the end endpoint when retrieving unused segment intervals.
+ * TODO: link GitHub issue
+ */
+ @Ignore
+ @Test
+ public void testKillEternitySegment()
{
- Assert.assertEquals(expectedDatasourceToLastKillIntervalEnd, target.getDatasourceToLastKillIntervalEnd());
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
+
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
Review Comment:
Constructing a `KillUnusedSegments` can be commoned out into a method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499377527
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
Review Comment:
Please remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499331990
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -34,33 +36,69 @@
import org.joda.time.Interval;
import javax.annotation.Nullable;
+import java.util.ArrayList;
import java.util.Collection;
+import java.util.Comparator;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
public class TestSegmentsMetadataManager implements SegmentsMetadataManager
{
+ private static final Logger log = new Logger(TestSegmentsMetadataManager.class);
+
+ /**
+ * segments = usedSegments + unusedSegments. Tracking unusedSegments separately for convenience and
+ * additional metadata.
+ */
private final ConcurrentMap<String, DataSegment> segments = new ConcurrentHashMap<>();
Review Comment:
Nit: The comment above doesn't really seem to add a lot of value. You could just call this field `allSegments` and be done with it.
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -34,33 +36,69 @@
import org.joda.time.Interval;
import javax.annotation.Nullable;
+import java.util.ArrayList;
import java.util.Collection;
+import java.util.Comparator;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
public class TestSegmentsMetadataManager implements SegmentsMetadataManager
{
+ private static final Logger log = new Logger(TestSegmentsMetadataManager.class);
+
+ /**
+ * segments = usedSegments + unusedSegments. Tracking unusedSegments separately for convenience and
+ * additional metadata.
+ */
private final ConcurrentMap<String, DataSegment> segments = new ConcurrentHashMap<>();
private final ConcurrentMap<String, DataSegment> usedSegments = new ConcurrentHashMap<>();
+ private final ConcurrentMap<String, DataSegmentPlus> unusedSegments = new ConcurrentHashMap<>();
private volatile DataSourcesSnapshot snapshot;
- public void addSegment(DataSegment segment)
+ /**
+ * Adds the used segment.
Review Comment:
Doesn't add any new info.
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -282,41 +300,16 @@ private Interval findIntervalForKill(String dataSource, DateTime maxUsedStatusLa
private int getAvailableKillTaskSlots(int killTaskCapacity, int numActiveKillTasks)
{
- return Math.max(
- 0,
- killTaskCapacity - numActiveKillTasks
- );
+ return Math.max(0, killTaskCapacity - numActiveKillTasks);
}
private boolean canDutyRun()
{
return lastKillTime == null || !DateTimes.nowUtc().isBefore(lastKillTime.plus(period));
}
- @VisibleForTesting
- static int getKillTaskCapacity(int totalWorkerCapacity, double killTaskSlotRatio, int maxKillTaskSlots)
+ private static int getKillTaskCapacity(int totalWorkerCapacity, double killTaskSlotRatio, int maxKillTaskSlots)
Review Comment:
Nit: If this is not visible anymore, does it still need to be static?
This method can be simplified by passing only `CoordinatorDynamicConfig` to it.
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +154,140 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
Collection<String> dataSourcesToKill =
params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
+ final double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
+ final int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
+ final int killTaskCapacity = getKillTaskCapacity(
CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
killTaskSlotRatio,
maxKillTaskSlots
);
- int availableKillTaskSlots = getAvailableKillTaskSlots(
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(
killTaskCapacity,
CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
);
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
-
if (0 < availableKillTaskSlots) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
+
+ stats.add(Stats.Kill.AVAILABLE_SLOTS, availableKillTaskSlots);
+ stats.add(Stats.Kill.MAX_SLOTS, killTaskCapacity);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
- )
- {
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
-
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
+ /**
+ * Spawn kill tasks for each datasource to be killed upto {@code availableKillTaskSlots}.
+ * @param dataSourcesToKill finalized set of datasources to kill
+ * @param availableKillTaskSlots number of available kill task slots
Review Comment:
Not adding any new info. Not really needed as this is just a private method.
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
Review Comment:
Please remove this.
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -44,50 +53,49 @@
import org.joda.time.Period;
import org.junit.Assert;
import org.junit.Before;
+import org.junit.Ignore;
import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.mockito.Answers;
-import org.mockito.ArgumentMatchers;
-import org.mockito.Mock;
-import org.mockito.Mockito;
-import org.mockito.junit.MockitoJUnitRunner;
+import javax.annotation.Nullable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.stream.Collectors;
+import java.util.Set;
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyInt;
-import static org.mockito.ArgumentMatchers.anyString;
/**
- *
+ * Unit tests to test the {@link KillUnusedSegments} duty.
*/
-@RunWith(MockitoJUnitRunner.class)
public class KillUnusedSegmentsTest
{
- private static final Duration INDEXING_PERIOD = Duration.standardSeconds(1);
- private static final Duration COORDINATOR_KILL_PERIOD = Duration.standardSeconds(1);
- private static final Duration DURATION_TO_RETAIN = Duration.standardDays(1);
+ private static final Logger log = new Logger(KillUnusedSegmentsTest.class);
+ private static final Duration INDEXING_PERIOD = Duration.standardSeconds(0);
+ private static final Duration COORDINATOR_KILL_PERIOD = Duration.standardSeconds(0);
+ private static final Duration DURATION_TO_RETAIN = Duration.standardHours(36);
+ private static final Duration BUFFER_PERIOD = Duration.standardSeconds(1);
private static final int MAX_SEGMENTS_TO_KILL = 10;
- private static final String DATASOURCE = "DS1";
-
- @Mock
- private SegmentsMetadataManager segmentsMetadataManager;
- @Mock
- private OverlordClient overlordClient;
- @Mock(answer = Answers.RETURNS_DEEP_STUBS)
- private DruidCoordinatorConfig config;
-
- @Mock
- private CoordinatorRunStats stats;
- @Mock
- private DruidCoordinatorRuntimeParams params;
- @Mock
- private CoordinatorDynamicConfig coordinatorDynamicConfig;
+ private static final String DS1 = "DS1";
+ private static final String DS2 = "DS2";
+
+ private TestSegmentsMetadataManager segmentsMetadataManager;
+
+ private TestOverlordClient overlordClient;
+
+ private TestDruidCoordinatorConfig.Builder configBuilder;
+
+ private DruidCoordinatorRuntimeParams.Builder paramsBuilder;
+
+ private final CoordinatorDynamicConfig.Builder dynamicConfigBuilder = CoordinatorDynamicConfig.builder();
+
+ private static final DateTime NOW = DateTimes.nowUtc();
+ private static final Interval YEAR_OLD = new Interval(Period.days(1), NOW.minusDays(365));
Review Comment:
More apt to call these `YEAR_TILL_DATE`, `MONTH_TILL_DATE` or something. `YEAR_OLD` is ambiguous. Also, all the periods are currently set to a day.
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +154,140 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
Review Comment:
Thanks for getting rid of this!
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -34,33 +36,69 @@
import org.joda.time.Interval;
import javax.annotation.Nullable;
+import java.util.ArrayList;
import java.util.Collection;
+import java.util.Comparator;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
public class TestSegmentsMetadataManager implements SegmentsMetadataManager
{
+ private static final Logger log = new Logger(TestSegmentsMetadataManager.class);
+
+ /**
+ * segments = usedSegments + unusedSegments. Tracking unusedSegments separately for convenience and
+ * additional metadata.
+ */
private final ConcurrentMap<String, DataSegment> segments = new ConcurrentHashMap<>();
private final ConcurrentMap<String, DataSegment> usedSegments = new ConcurrentHashMap<>();
+ private final ConcurrentMap<String, DataSegmentPlus> unusedSegments = new ConcurrentHashMap<>();
private volatile DataSourcesSnapshot snapshot;
- public void addSegment(DataSegment segment)
+ /**
+ * Adds the used segment.
+ */
+ public void addSegment(DataSegment segment, boolean isUsed)
{
segments.put(segment.getId().toString(), segment);
usedSegments.put(segment.getId().toString(), segment);
snapshot = null;
}
+ /**
+ * Remove the used segment.
+ */
public void removeSegment(DataSegment segment)
{
segments.remove(segment.getId().toString());
usedSegments.remove(segment.getId().toString());
snapshot = null;
}
+ /**
+ * Add the unused segment with metadata.
Review Comment:
I don't think any of these comments are really needed. The methods are fairly self-explanatory.
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
Review Comment:
Just calling this out in case it got missed in the refactor.
The original segments had their _end_ a year ago from now, a month ago from now and so on.
The new intervals `YEAR_OLD`, `MONTH_OLD` actually have their end at now.
Please ensure that the tests have been updated to incorporate that.
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -34,33 +36,69 @@
import org.joda.time.Interval;
import javax.annotation.Nullable;
+import java.util.ArrayList;
import java.util.Collection;
+import java.util.Comparator;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
public class TestSegmentsMetadataManager implements SegmentsMetadataManager
{
+ private static final Logger log = new Logger(TestSegmentsMetadataManager.class);
+
+ /**
+ * segments = usedSegments + unusedSegments. Tracking unusedSegments separately for convenience and
+ * additional metadata.
+ */
private final ConcurrentMap<String, DataSegment> segments = new ConcurrentHashMap<>();
private final ConcurrentMap<String, DataSegment> usedSegments = new ConcurrentHashMap<>();
+ private final ConcurrentMap<String, DataSegmentPlus> unusedSegments = new ConcurrentHashMap<>();
private volatile DataSourcesSnapshot snapshot;
- public void addSegment(DataSegment segment)
+ /**
+ * Adds the used segment.
+ */
+ public void addSegment(DataSegment segment, boolean isUsed)
Review Comment:
Where is the boolean `isUsed` used?
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -221,7 +265,28 @@ public List<Interval> getUnusedSegmentIntervals(
final DateTime maxUsedStatusLastUpdatedTime
)
{
- return null;
+ final List<DataSegmentPlus> sortedDataSegments = new ArrayList<>(unusedSegments.values());
+ sortedDataSegments.sort(
+ Comparator.comparingLong(dataSegmentPlus -> dataSegmentPlus.getDataSegment().getInterval().getStartMillis()
+ )
Review Comment:
Please use consistent styles:
```suggestion
Comparator.comparingLong(
dataSegmentPlus -> dataSegmentPlus.getDataSegment().getInterval().getStartMillis()
)
```
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +154,140 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
Collection<String> dataSourcesToKill =
params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
+ final double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
+ final int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
+ final int killTaskCapacity = getKillTaskCapacity(
CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
killTaskSlotRatio,
maxKillTaskSlots
);
- int availableKillTaskSlots = getAvailableKillTaskSlots(
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(
killTaskCapacity,
CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
);
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
-
if (0 < availableKillTaskSlots) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
+
+ stats.add(Stats.Kill.AVAILABLE_SLOTS, availableKillTaskSlots);
+ stats.add(Stats.Kill.MAX_SLOTS, killTaskCapacity);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
- )
- {
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
-
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
+ /**
+ * Spawn kill tasks for each datasource to be killed upto {@code availableKillTaskSlots}.
+ * @param dataSourcesToKill finalized set of datasources to kill
+ * @param availableKillTaskSlots number of available kill task slots
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
Review Comment:
Much easier to read now, thanks!
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
Review Comment:
Please update this test name to indicate whether it tests only the default config values or the default behaviour of the duty.
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +154,140 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
Collection<String> dataSourcesToKill =
params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
+ final double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
+ final int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
+ final int killTaskCapacity = getKillTaskCapacity(
CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
killTaskSlotRatio,
maxKillTaskSlots
);
- int availableKillTaskSlots = getAvailableKillTaskSlots(
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(
killTaskCapacity,
CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
);
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
-
if (0 < availableKillTaskSlots) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
+
+ stats.add(Stats.Kill.AVAILABLE_SLOTS, availableKillTaskSlots);
+ stats.add(Stats.Kill.MAX_SLOTS, killTaskCapacity);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
- )
- {
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
-
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
+ /**
+ * Spawn kill tasks for each datasource to be killed upto {@code availableKillTaskSlots}.
+ * @param dataSourcesToKill finalized set of datasources to kill
+ * @param availableKillTaskSlots number of available kill task slots
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
+ log.info("Submitted [%d] kill tasks for [%d] datasources.%s",
submittedTasks,
dataSourcesToKill.size(),
availableKillTaskSlots < dataSourcesToKill.size()
? StringUtils.format(
" Datasources skipped: %s",
- ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size())
- )
+ ImmutableList.copyOf(dataSourcesToKill).subList(submittedTasks, dataSourcesToKill.size()))
: ""
- );
Review Comment:
This style of determining skipped datasources is extremely convoluted and that too just for logging.
It would be better to just keep a set of `remainingDatasourcesToKill` and keep removing items from it as soon as tasks are submitted for that datasource.
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/KillUnusedSegments.java:
##########
@@ -146,130 +154,140 @@ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)
return runInternal(params);
}
- @VisibleForTesting
- DruidCoordinatorRuntimeParams runInternal(DruidCoordinatorRuntimeParams params)
+ private DruidCoordinatorRuntimeParams runInternal(final DruidCoordinatorRuntimeParams params)
{
- TaskStats taskStats = new TaskStats();
Collection<String> dataSourcesToKill =
params.getCoordinatorDynamicConfig().getSpecificDataSourcesToKillUnusedSegmentsIn();
- double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
- int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
- int killTaskCapacity = getKillTaskCapacity(
+ final double killTaskSlotRatio = params.getCoordinatorDynamicConfig().getKillTaskSlotRatio();
+ final int maxKillTaskSlots = params.getCoordinatorDynamicConfig().getMaxKillTaskSlots();
+ final int killTaskCapacity = getKillTaskCapacity(
CoordinatorDutyUtils.getTotalWorkerCapacity(overlordClient),
killTaskSlotRatio,
maxKillTaskSlots
);
- int availableKillTaskSlots = getAvailableKillTaskSlots(
+ final int availableKillTaskSlots = getAvailableKillTaskSlots(
killTaskCapacity,
CoordinatorDutyUtils.getNumActiveTaskSlots(overlordClient, IS_AUTO_KILL_TASK).size()
);
final CoordinatorRunStats stats = params.getCoordinatorStats();
- taskStats.availableTaskSlots = availableKillTaskSlots;
- taskStats.maxSlots = killTaskCapacity;
-
if (0 < availableKillTaskSlots) {
// If no datasource has been specified, all are eligible for killing unused segments
if (CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
dataSourcesToKill = segmentsMetadataManager.retrieveAllDataSourceNames();
}
- log.debug("Killing unused segments for datasources[%s]", dataSourcesToKill);
lastKillTime = DateTimes.nowUtc();
- taskStats.submittedTasks = killUnusedSegments(dataSourcesToKill, availableKillTaskSlots);
+ killUnusedSegments(dataSourcesToKill, availableKillTaskSlots, stats);
}
// any datasources that are no longer being considered for kill should have their
// last kill interval removed from map.
datasourceToLastKillIntervalEnd.keySet().retainAll(dataSourcesToKill);
- addStats(taskStats, stats);
+
+ stats.add(Stats.Kill.AVAILABLE_SLOTS, availableKillTaskSlots);
+ stats.add(Stats.Kill.MAX_SLOTS, killTaskCapacity);
return params;
}
- private void addStats(
- TaskStats taskStats,
- CoordinatorRunStats stats
- )
- {
- stats.add(Stats.Kill.AVAILABLE_SLOTS, taskStats.availableTaskSlots);
- stats.add(Stats.Kill.SUBMITTED_TASKS, taskStats.submittedTasks);
- stats.add(Stats.Kill.MAX_SLOTS, taskStats.maxSlots);
- }
-
- private int killUnusedSegments(
- Collection<String> dataSourcesToKill,
- int availableKillTaskSlots
+ /**
+ * Spawn kill tasks for each datasource to be killed upto {@code availableKillTaskSlots}.
+ * @param dataSourcesToKill finalized set of datasources to kill
+ * @param availableKillTaskSlots number of available kill task slots
+ */
+ private void killUnusedSegments(
+ @Nullable final Collection<String> dataSourcesToKill,
+ final int availableKillTaskSlots,
+ final CoordinatorRunStats stats
)
{
+ if (CollectionUtils.isNullOrEmpty(dataSourcesToKill) || availableKillTaskSlots <= 0) {
+ stats.add(Stats.Kill.SUBMITTED_TASKS, 0);
+ return;
+ }
int submittedTasks = 0;
- if (0 < availableKillTaskSlots && !CollectionUtils.isNullOrEmpty(dataSourcesToKill)) {
- for (String dataSource : dataSourcesToKill) {
- if (submittedTasks >= availableKillTaskSlots) {
- log.debug(StringUtils.format(
- "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
- + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
- break;
- }
- final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
- final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime);
- if (intervalToKill == null) {
- datasourceToLastKillIntervalEnd.remove(dataSource);
- continue;
- }
+ for (String dataSource : dataSourcesToKill) {
+ if (submittedTasks >= availableKillTaskSlots) {
+ log.info(StringUtils.format(
+ "Submitted [%d] kill tasks and reached kill task slot limit [%d]. Will resume "
+ + "on the next coordinator cycle.", submittedTasks, availableKillTaskSlots));
+ break;
+ }
+ final DateTime maxUsedStatusLastUpdatedTime = DateTimes.nowUtc().minus(bufferPeriod);
+ final Interval intervalToKill = findIntervalForKill(dataSource, maxUsedStatusLastUpdatedTime, stats);
+ if (intervalToKill == null) {
+ datasourceToLastKillIntervalEnd.remove(dataSource);
+ continue;
+ }
- try {
- FutureUtils.getUnchecked(
- overlordClient.runKillTask(
- TASK_ID_PREFIX,
- dataSource,
- intervalToKill,
- maxSegmentsToKill,
- maxUsedStatusLastUpdatedTime
- ),
- true
- );
- ++submittedTasks;
- datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
- }
- catch (Exception ex) {
- log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
- if (Thread.currentThread().isInterrupted()) {
- log.warn("Skipping kill task scheduling because thread is interrupted.");
- break;
- }
+ try {
+ FutureUtils.getUnchecked(
+ overlordClient.runKillTask(
+ TASK_ID_PREFIX,
+ dataSource,
+ intervalToKill,
+ maxSegmentsToKill,
+ maxUsedStatusLastUpdatedTime
+ ),
+ true
+ );
+ ++submittedTasks;
+ datasourceToLastKillIntervalEnd.put(dataSource, intervalToKill.getEnd());
+ }
+ catch (Exception ex) {
+ log.error(ex, "Failed to submit kill task for dataSource[%s] in interval[%s]", dataSource, intervalToKill);
+ if (Thread.currentThread().isInterrupted()) {
+ log.warn("Skipping kill task scheduling because thread is interrupted.");
+ break;
}
}
}
- if (log.isDebugEnabled()) {
- log.debug(
- "Submitted [%d] kill tasks for [%d] datasources.%s",
+ log.info("Submitted [%d] kill tasks for [%d] datasources.%s",
Review Comment:
The original style was more consistent:
```suggestion
log.info(
"Submitted [%d] kill tasks for [%d] datasources.%s",
```
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
+ */
+ @Test
+ public void testRunsWithKillPeriod()
+ {
+ configBuilder.withCoordinatorKillPeriod(Duration.standardHours(1));
+
+ setupUnusedSegments();
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
+ );
+
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ secondRun.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
+ }
+
+ /**
+ * Similar to {@link #testMultipleRuns()}
+ */
+ @Test
+ public void testMultipleDatasources()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, NEXT_DAY, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 2, 10, ImmutableMap.of(DS1, 2L, DS2, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ validateAndResetState(DS2, new Interval(YEAR_OLD.getStart(), DAY_OLD.getEnd()));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 4, 20, ImmutableMap.of(DS1, 4L, DS2, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+ validateAndResetState(DS2, NEXT_DAY);
+
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 5, 30, ImmutableMap.of(DS1, 5L, DS2, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * Even though "more recent" segments are also considered as candidates in the wide kill interval here,
+ * the kill task will narrow down to clean up only the segments that strictly honor the max kill time.
+ * See {@code KillUnusedSegmentsTaskTest#testKillMultipleUnusedSegmentsWithNullMaxUsedStatusLastUpdatedTime}
+ * for example.
+ */
+ @Test
+ public void testSpreadOutLastMaxSegments()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillBufferPeriod(Duration.standardDays(3));
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, HOUR_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(10));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 4L)),
+ firstKill.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), NEXT_MONTH.getEnd()));
+ }
+
+ @Test
+ public void testAddOldVersionsLater()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+
+ log.info("Second run...");
+ // Add two old unused segments now. These only get killed much later on when the kill
+ // duty eventually round robins its way through until the latest time.
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 2, 30, ImmutableMap.of(DS1, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+
+ log.info("Fourth run...");
Review Comment:
Please remove these log lines once the tests are settled. The duty itself would be emitting logs.
##########
server/src/test/java/org/apache/druid/server/coordinator/simulate/TestSegmentsMetadataManager.java:
##########
@@ -221,7 +265,28 @@ public List<Interval> getUnusedSegmentIntervals(
final DateTime maxUsedStatusLastUpdatedTime
)
{
- return null;
+ final List<DataSegmentPlus> sortedDataSegments = new ArrayList<>(unusedSegments.values());
Review Comment:
You could also just use a `TreeSet`.
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
- Mockito.doReturn(stats).when(params).getCoordinatorStats();
- Mockito.doReturn(COORDINATOR_KILL_PERIOD).when(config).getCoordinatorKillPeriod();
- Mockito.doReturn(DURATION_TO_RETAIN).when(config).getCoordinatorKillDurationToRetain();
- Mockito.doReturn(INDEXING_PERIOD).when(config).getCoordinatorIndexingPeriod();
- Mockito.doReturn(MAX_SEGMENTS_TO_KILL).when(config).getCoordinatorKillMaxSegments();
- Mockito.doReturn(Duration.parse("PT3154000000S")).when(config).getCoordinatorKillBufferPeriod();
+ segmentsMetadataManager = new TestSegmentsMetadataManager();
+ overlordClient = new TestOverlordClient();
- Mockito.doReturn(Collections.singleton(DATASOURCE))
- .when(coordinatorDynamicConfig).getSpecificDataSourcesToKillUnusedSegmentsIn();
+ // These two can definitely be part of setup()
+ configBuilder = new TestDruidCoordinatorConfig.Builder()
+ .withCoordinatorIndexingPeriod(INDEXING_PERIOD)
+ .withCoordinatorKillPeriod(COORDINATOR_KILL_PERIOD)
+ .withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN)
+ .withCoordinatorKillMaxSegments(MAX_SEGMENTS_TO_KILL)
+ .withCoordinatorKillBufferPeriod(BUFFER_PERIOD);
+ paramsBuilder = DruidCoordinatorRuntimeParams.newBuilder(DateTimes.nowUtc());
+ }
- final DateTime now = DateTimes.nowUtc();
+ /**
+ * The buffer periood and duration to retain influence kill behavior.
+ */
+ @Test
+ public void testDefaults()
+ {
+ final DateTime sixtyDaysAgo = NOW.minusDays(60);
- yearOldSegment = createSegmentWithEnd(now.minusDays(365));
- monthOldSegment = createSegmentWithEnd(now.minusDays(30));
- dayOldSegment = createSegmentWithEnd(now.minusDays(1));
- hourOldSegment = createSegmentWithEnd(now.minusHours(1));
- nextDaySegment = createSegmentWithEnd(now.plusDays(1));
- nextMonthSegment = createSegmentWithEnd(now.plusDays(30));
-
- final List<DataSegment> unusedSegments = ImmutableList.of(
- yearOldSegment,
- monthOldSegment,
- dayOldSegment,
- hourOldSegment,
- nextDaySegment,
- nextMonthSegment
- );
-
- Mockito.when(
- segmentsMetadataManager.getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
- )
- ).thenAnswer(invocation -> {
- DateTime minStartTime = invocation.getArgument(1);
- DateTime maxEndTime = invocation.getArgument(2);
- long maxEndMillis = maxEndTime.getMillis();
- Long minStartMillis = minStartTime != null ? minStartTime.getMillis() : null;
- List<Interval> unusedIntervals =
- unusedSegments.stream()
- .map(DataSegment::getInterval)
- .filter(i -> i.getEnd().getMillis() <= maxEndMillis
- && (null == minStartMillis || i.getStart().getMillis() >= minStartMillis))
- .collect(Collectors.toList());
-
- int limit = invocation.getArgument(3);
- return unusedIntervals.size() <= limit ? unusedIntervals : unusedIntervals.subList(0, limit);
- });
-
- target = new KillUnusedSegments(
+ createAndAddUnusedSegment(DS1, YEAR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, MONTH_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, DAY_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, HOUR_OLD, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_DAY, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, sixtyDaysAgo);
+ createAndAddUnusedSegment(DS1, Intervals.ETERNITY, sixtyDaysAgo);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ new TestDruidCoordinatorConfig.Builder().build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ firstRun.getCoordinatorStats()
);
+
+ validateAndResetState(DS1, YEAR_OLD);
}
@Test
public void testRunWithNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(null).when(segmentsMetadataManager).getUnusedSegmentIntervals(
- ArgumentMatchers.anyString(),
- ArgumentMatchers.any(),
- ArgumentMatchers.any(),
- ArgumentMatchers.anyInt(),
- ArgumentMatchers.any()
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
}
@Test
public void testRunWithSpecificDatasourceAndNoIntervalShouldNotKillAnySegments()
{
- Mockito.doReturn(Duration.standardDays(400))
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(Duration.standardDays(400));
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
);
- // No unused segment is older than the retention period
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- target.run(params);
- Mockito.verify(overlordClient, Mockito.never())
- .runKillTask(anyString(), anyString(), any(Interval.class), anyInt(), any(DateTime.class));
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * The kill period is honored after the first indexing run.
+ */
+ @Test
+ public void testRunsWithKillPeriod()
+ {
+ configBuilder.withCoordinatorKillPeriod(Duration.standardHours(1));
+
+ setupUnusedSegments();
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
+ );
+
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ secondRun.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
+ }
+
+ /**
+ * Similar to {@link #testMultipleRuns()}
+ */
+ @Test
+ public void testMultipleDatasources()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, NEXT_DAY, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 2, 10, ImmutableMap.of(DS1, 2L, DS2, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ validateAndResetState(DS2, new Interval(YEAR_OLD.getStart(), DAY_OLD.getEnd()));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 4, 20, ImmutableMap.of(DS1, 4L, DS2, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+ validateAndResetState(DS2, NEXT_DAY);
+
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 5, 30, ImmutableMap.of(DS1, 5L, DS2, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+ validateAndResetState(DS2, null);
+ }
+
+ /**
+ * Even though "more recent" segments are also considered as candidates in the wide kill interval here,
+ * the kill task will narrow down to clean up only the segments that strictly honor the max kill time.
+ * See {@code KillUnusedSegmentsTaskTest#testKillMultipleUnusedSegmentsWithNullMaxUsedStatusLastUpdatedTime}
+ * for example.
+ */
+ @Test
+ public void testSpreadOutLastMaxSegments()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillBufferPeriod(Duration.standardDays(3));
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, HOUR_OLD, NOW.minusDays(2));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(10));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(10));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 4L)),
+ firstKill.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), NEXT_MONTH.getEnd()));
+ }
+
+ @Test
+ public void testAddOldVersionsLater()
+ {
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ createAndAddUnusedSegment(DS1, DAY_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_DAY, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, NEXT_MONTH, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams firstKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(DAY_OLD.getStart(), NEXT_DAY.getEnd()));
+
+ log.info("Second run...");
+ // Add two old unused segments now. These only get killed much later on when the kill
+ // duty eventually round robins its way through until the latest time.
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS1, MONTH_OLD, NOW.minusDays(1));
+
+ final DruidCoordinatorRuntimeParams secondKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 3L)),
+ secondKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, NEXT_MONTH);
+
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(30, 2, 30, ImmutableMap.of(DS1, 3L)),
+ thirdKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+
+ log.info("Fourth run...");
+
+ final DruidCoordinatorRuntimeParams fourthKill = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(40, 3, 40, ImmutableMap.of(DS1, 5L)),
+ fourthKill.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, new Interval(YEAR_OLD.getStart(), MONTH_OLD.getEnd()));
+ }
+
+ @Test
+ public void testNoDatasources()
+ {
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 0, 10, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS1, null);
+ validateAndResetState(DS2, null);
+ }
+
+ @Test
+ public void testWhiteList()
+ {
+ // Only segments more than a day old are killed
+ dynamicConfigBuilder.withSpecificDataSourcesToKillUnusedSegmentsIn(Collections.singleton(DS2));
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ createAndAddUnusedSegment(DS1, YEAR_OLD, NOW.minusDays(1));
+ createAndAddUnusedSegment(DS2, YEAR_OLD, NOW.minusDays(1));
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 0L, DS2, 1L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(DS2, YEAR_OLD);
+ validateAndResetState(DS1, null);
}
@Test
public void testDurationToRetain()
{
// Only segments more than a day old are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- dayOldSegment.getInterval().getEnd()
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ runParams.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(), monthOldSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, dayOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testNegativeDurationToRetain()
{
- // Duration to retain = -1 day, reinit target for config to take effect
- Mockito.doReturn(DURATION_TO_RETAIN.negated())
- .when(config).getCoordinatorKillDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillDurationToRetain(DURATION_TO_RETAIN.negated());
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 5L)),
+ runParams.getCoordinatorStats()
);
- // Segments upto 1 day in the future are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextDaySegment.getInterval().getEnd()
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextDaySegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextDaySegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testIgnoreDurationToRetain()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 6L)),
+ runParams.getCoordinatorStats()
);
- // All future and past unused segments are killed
- Interval expectedKillInterval = new Interval(
- yearOldSegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
+ // All past and future unused segments should be killed
+ validateAndResetState(
+ DS1,
+ new Interval(yearOldSegment.getInterval().getStart(), nextMonthSegment.getInterval().getEnd())
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(expectedKillInterval);
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
}
@Test
public void testMaxSegmentsToKill()
{
- Mockito.doReturn(1)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillMaxSegments(1);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 1L)),
+ runParams.getCoordinatorStats()
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- // Only 1 unused segment is killed
- runAndVerifyKillInterval(yearOldSegment.getInterval());
- verifyState(ImmutableMap.of(DATASOURCE, yearOldSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10);
+ validateAndResetState(
+ DS1,
+ yearOldSegment.getInterval()
+ );
}
@Test
public void testMultipleRuns()
{
- Mockito.doReturn(true)
- .when(config).getCoordinatorKillIgnoreDurationToRetain();
- Mockito.doReturn(2)
- .when(config).getCoordinatorKillMaxSegments();
- target = new KillUnusedSegments(
+ configBuilder.withCoordinatorKillIgnoreDurationToRetain(true);
+ configBuilder.withCoordinatorKillMaxSegments(2);
+
+ setupUnusedSegments();
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
segmentsMetadataManager,
overlordClient,
- config
+ configBuilder.build()
+ );
+
+ log.info("First run...");
+ final DruidCoordinatorRuntimeParams firstRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(10, 1, 10, ImmutableMap.of(DS1, 2L)),
+ firstRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ yearOldSegment.getInterval().getStart(),
+ monthOldSegment.getInterval().getEnd()
+ )
);
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- yearOldSegment.getInterval().getStart(),
- monthOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, monthOldSegment.getInterval().getEnd()));
+ log.info("Second run...");
+ final DruidCoordinatorRuntimeParams secondRun = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(20, 2, 20, ImmutableMap.of(DS1, 4L)),
+ secondRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ dayOldSegment.getInterval().getStart(),
+ hourOldSegment.getInterval().getEnd()
+ )
+ );
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- dayOldSegment.getInterval().getStart(),
- hourOldSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, hourOldSegment.getInterval().getEnd()));
+ log.info("Third run...");
+ final DruidCoordinatorRuntimeParams thirdRun = killDuty.run(paramsBuilder.build());
- mockTaskSlotUsage(1.0, Integer.MAX_VALUE, 1, 10);
- runAndVerifyKillInterval(new Interval(
- nextDaySegment.getInterval().getStart(),
- nextMonthSegment.getInterval().getEnd()
- ));
- verifyState(ImmutableMap.of(DATASOURCE, nextMonthSegment.getInterval().getEnd()));
- verifyStats(9, 1, 10, 3);
+ validateStats(
+ new ExpectedStats(30, 3, 30, ImmutableMap.of(DS1, 6L)),
+ thirdRun.getCoordinatorStats()
+ );
+
+ validateAndResetState(
+ DS1,
+ new Interval(
+ nextDaySegment.getInterval().getStart(),
+ nextMonthSegment.getInterval().getEnd()
+ )
+ );
}
@Test
public void testKillTaskSlotRatioNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(0.10, 10, 1, 5);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 0);
+ dynamicConfigBuilder.withKillTaskSlotRatio(0.10);
+ dynamicConfigBuilder.withMaxKillTaskSlots(10);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(1, 5);
+
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 0, ImmutableMap.of()),
+ runParams.getCoordinatorStats()
+ );
+ validateAndResetState(DS1, null);
}
@Test
public void testMaxKillTaskSlotsNoAvailableTaskCapacityForKill()
{
- mockTaskSlotUsage(1.0, 3, 3, 10);
- runAndVerifyNoKill();
- verifyState(ImmutableMap.of());
- verifyStats(0, 0, 3);
+ dynamicConfigBuilder.withKillTaskSlotRatio(1.0);
+ dynamicConfigBuilder.withMaxKillTaskSlots(3);
+ paramsBuilder.withDynamicConfigs(dynamicConfigBuilder.build());
+
+ overlordClient = new TestOverlordClient(3, 10);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+ overlordClient.addTask(DS1);
+
+ final KillUnusedSegments killDuty = new KillUnusedSegments(
+ segmentsMetadataManager,
+ overlordClient,
+ configBuilder.build()
+ );
+
+ final DruidCoordinatorRuntimeParams runParams = killDuty.run(paramsBuilder.build());
+
+ validateStats(
+ new ExpectedStats(0, 0, 3, ImmutableMap.of()),
Review Comment:
I feel it would be much more readable to just validate the relevant stats individually, rather than having to pass zeroes and empty maps repeatedly. This would allow someone reading the test to instantly recognize exactly what is being verified.
You can see some examples in `RunRulesTest`.
Something like:
```
Assert.assertEquals(10L, stats.getSegmentStat(Stats.Segments.ASSIGNED, "normal", DATASOURCE));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] WIP: Auto-kill bug fixes and improvements (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15941:
URL: https://github.com/apache/druid/pull/15941#discussion_r1499401531
##########
server/src/test/java/org/apache/druid/server/coordinator/duty/KillUnusedSegmentsTest.java:
##########
@@ -96,362 +104,805 @@ public class KillUnusedSegmentsTest
private DataSegment nextDaySegment;
private DataSegment nextMonthSegment;
- private KillUnusedSegments target;
@Before
public void setup()
{
- Mockito.doReturn(coordinatorDynamicConfig).when(params).getCoordinatorDynamicConfig();
Review Comment:
Nice to get rid of all the unncessary mocks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org