You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2026/06/01 11:13:52 UTC
[PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
clintropolis opened a new pull request, #19535:
URL: https://github.com/apache/druid/pull/19535
### Description
changes:
* adds `acquirePartialSegment` / `acquireCachedPartialSegment` to `SegmentCacheManager` and `SegmentManager` to allow callers to opt-in to async partial segments; MSQ `RegularLoadableSegment` uses the new partial path
* `PartialSegmentMetadataCacheEntry`, `PartialSegmentBundleCacheEntry` are now wired into `SegmentLocalCacheManager`, along with added `PartialBundleAcquirer` helper to pass things like download thread pool and the ability to acquire reference holds on cache entrie
* adds `PartialQueryableIndexSegment`, `PartialQueryableIndexCursorFactory`, `V10TimeBoundaryInspector` for references acquired from the now wired up metadata cache entry, implementing the async cursor holder contract
* moved `DirectoryBackedRangeReader` out of tests to use as the implementation for `LocalLoadSpec` range reader.
* add `FilteredCursorFactory`/`RestrictedCursorFactory` async cursor implementations
* adds config `druid.segmentCache.virtualStoragePartialDownloadsEnabled to enable feature, on iff druid.segmentCache.virtualStorage is on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3343889417
##########
processing/src/test/java/org/apache/druid/segment/PartialQueryableIndexCursorFactoryTest.java:
##########
@@ -0,0 +1,722 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.ForwardingListeningExecutorService;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import com.google.common.util.concurrent.SettableFuture;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.ListBasedInputRow;
+import org.apache.druid.data.input.impl.AggregateProjectionSpec;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.LongDimensionSchema;
+import org.apache.druid.data.input.impl.StringDimensionSchema;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import org.apache.druid.query.filter.EqualityFilter;
+import org.apache.druid.segment.column.ColumnConfig;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.CompressionStrategy;
+import org.apache.druid.segment.file.CountingRangeReader;
+import org.apache.druid.segment.file.PartialSegmentFileMapperV10;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+class PartialQueryableIndexCursorFactoryTest extends InitializedNullHandlingTest
+{
+ private static final ColumnConfig COLUMN_CONFIG = ColumnConfig.DEFAULT;
+ private static final DateTime TIME = DateTimes.of("2025-01-01");
+ private static final String PROJECTION_NAME = "dim1_metric1_sum";
+
+ private static final RowSignature ROW_SIGNATURE = RowSignature.builder()
+ .add("dim1", ColumnType.STRING)
+ .add("metric1", ColumnType.LONG)
+ .build();
+
+ private static final List<AggregateProjectionSpec> PROJECTIONS = Collections.singletonList(
+ AggregateProjectionSpec.builder(PROJECTION_NAME)
+ .groupingColumns(new StringDimensionSchema("dim1"))
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+
+ private static final List<InputRow> ROWS = Arrays.asList(
+ new ListBasedInputRow(ROW_SIGNATURE, TIME, ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 1L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(1), ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 2L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(2), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 3L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(3), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 4L))
+ );
+
+ @TempDir
+ static File sharedTempDir;
+
+ private static File segmentDir;
+ private static ListeningExecutorService realExec;
+
+ @TempDir
+ File perTestTempDir;
+
+ @BeforeAll
+ static void buildSegment()
+ {
+ final File tmpDir = new File(sharedTempDir, "build_" + ThreadLocalRandom.current().nextInt());
+ segmentDir = IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmpDir)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(PROJECTIONS)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder().withMetadataCompression(CompressionStrategy.NONE).build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+
+ realExec = MoreExecutors.listeningDecorator(Execs.singleThreaded("partial-cursor-test-%d"));
+ }
+
+ @AfterAll
+ static void teardownExec()
+ {
+ realExec.shutdownNow();
+ }
+
+ private record IndexAndMapper(PartialQueryableIndex index, PartialSegmentFileMapperV10 mapper)
+ implements AutoCloseable
+ {
+ @Override
+ public void close()
+ {
+ mapper.close();
+ }
+ }
+
+ private IndexAndMapper openIndex(CountingRangeReader rangeReader, String cacheName) throws IOException
+ {
+ final File cacheDir = new File(perTestTempDir, cacheName);
+ FileUtils.mkdirp(cacheDir);
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ TestHelper.makeJsonMapper(),
+ cacheDir,
+ IndexIO.V10_FILE_NAME,
+ Collections.emptyList()
+ );
+ return new IndexAndMapper(
+ new PartialQueryableIndex(mapper.getSegmentFileMetadata(), mapper, COLUMN_CONFIG),
+ mapper
+ );
+ }
+
+ private static ListeningExecutorService directExec()
+ {
+ return MoreExecutors.listeningDecorator(MoreExecutors.newDirectExecutorService());
+ }
+
+ private static PartialBundleAcquirer noOpAcquirer(ListeningExecutorService downloadExec)
+ {
+ return new PartialBundleAcquirer()
+ {
+ @Override
+ public Closeable acquire(String bundleName)
+ {
+ return () -> {};
+ }
+
+ @Override
+ public ListeningExecutorService getDownloadExec()
+ {
+ return downloadExec;
+ }
+ };
+ }
+
+ @Test
+ void testSyncMakeCursorHolderThrowsWhenSegmentNotFullyDownloaded() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_throws")) {
+ final PartialQueryableIndex index = opened.index();
+ Assertions.assertFalse(index.isFullyDownloaded());
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Fresh mapper has only the header on disk; no internal files have been downloaded. The sync path must
+ // refuse rather than trigger downloads on the calling thread.
+ Assertions.assertThrows(
+ DruidException.class,
+ () -> factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)
+ );
+ }
+ }
+
+ @Test
+ void testSyncMakeCursorHolderSucceedsAfterFullDownload() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_after_async")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Eagerly materialize the entire segment via the file mapper to simulate the eager-acquire path.
+ mapper.ensureFilesAvailable(mapper.getSegmentFileMetadata().getFiles().keySet());
+ Assertions.assertTrue(mapper.isFullyDownloaded(), "test precondition: every internal file should be on disk");
+
+ try (CursorHolder holder = factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)) {
+ Assertions.assertNotNull(holder);
+ }
+ }
+ }
+
+ @Test
+ void testAsyncMakeCursorHolderDefersDownloadUntilExecutorRuns() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ final CountDownLatch gate = new CountDownLatch(1);
+ final ExecutorService rawExec = Execs.singleThreaded("partial-cursor-defer-%d");
+ final ListeningExecutorService gatedExec = MoreExecutors.listeningDecorator(rawExec);
+ try (IndexAndMapper opened = openIndex(rangeReader, "async_defer")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(gatedExec)
+ );
+ rangeReader.resetCount();
+
+ // Hold the executor with a pre-queued blocker so the actual download task can't start.
+ // assign the blocker future to an (intentionally unused) local so errorprone's CheckReturnValue is satisfied
+ @SuppressWarnings("unused")
+ ListenableFuture<?> unused = gatedExec.submit(() -> {
+ try {
+ gate.await();
+ }
+ catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ });
Review Comment:
## CodeQL / Unread local variable
Variable 'ListenableFuture<?> unused' is never read.
[Show more details](https://github.com/apache/druid/security/code-scanning/11287)
##########
processing/src/test/java/org/apache/druid/segment/PartialQueryableIndexCursorFactoryTest.java:
##########
@@ -0,0 +1,722 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.ForwardingListeningExecutorService;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import com.google.common.util.concurrent.SettableFuture;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.ListBasedInputRow;
+import org.apache.druid.data.input.impl.AggregateProjectionSpec;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.LongDimensionSchema;
+import org.apache.druid.data.input.impl.StringDimensionSchema;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import org.apache.druid.query.filter.EqualityFilter;
+import org.apache.druid.segment.column.ColumnConfig;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.CompressionStrategy;
+import org.apache.druid.segment.file.CountingRangeReader;
+import org.apache.druid.segment.file.PartialSegmentFileMapperV10;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+class PartialQueryableIndexCursorFactoryTest extends InitializedNullHandlingTest
+{
+ private static final ColumnConfig COLUMN_CONFIG = ColumnConfig.DEFAULT;
+ private static final DateTime TIME = DateTimes.of("2025-01-01");
+ private static final String PROJECTION_NAME = "dim1_metric1_sum";
+
+ private static final RowSignature ROW_SIGNATURE = RowSignature.builder()
+ .add("dim1", ColumnType.STRING)
+ .add("metric1", ColumnType.LONG)
+ .build();
+
+ private static final List<AggregateProjectionSpec> PROJECTIONS = Collections.singletonList(
+ AggregateProjectionSpec.builder(PROJECTION_NAME)
+ .groupingColumns(new StringDimensionSchema("dim1"))
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+
+ private static final List<InputRow> ROWS = Arrays.asList(
+ new ListBasedInputRow(ROW_SIGNATURE, TIME, ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 1L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(1), ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 2L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(2), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 3L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(3), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 4L))
+ );
+
+ @TempDir
+ static File sharedTempDir;
+
+ private static File segmentDir;
+ private static ListeningExecutorService realExec;
+
+ @TempDir
+ File perTestTempDir;
+
+ @BeforeAll
+ static void buildSegment()
+ {
+ final File tmpDir = new File(sharedTempDir, "build_" + ThreadLocalRandom.current().nextInt());
+ segmentDir = IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmpDir)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(PROJECTIONS)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder().withMetadataCompression(CompressionStrategy.NONE).build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+
+ realExec = MoreExecutors.listeningDecorator(Execs.singleThreaded("partial-cursor-test-%d"));
+ }
+
+ @AfterAll
+ static void teardownExec()
+ {
+ realExec.shutdownNow();
+ }
+
+ private record IndexAndMapper(PartialQueryableIndex index, PartialSegmentFileMapperV10 mapper)
+ implements AutoCloseable
+ {
+ @Override
+ public void close()
+ {
+ mapper.close();
+ }
+ }
+
+ private IndexAndMapper openIndex(CountingRangeReader rangeReader, String cacheName) throws IOException
+ {
+ final File cacheDir = new File(perTestTempDir, cacheName);
+ FileUtils.mkdirp(cacheDir);
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ TestHelper.makeJsonMapper(),
+ cacheDir,
+ IndexIO.V10_FILE_NAME,
+ Collections.emptyList()
+ );
+ return new IndexAndMapper(
+ new PartialQueryableIndex(mapper.getSegmentFileMetadata(), mapper, COLUMN_CONFIG),
+ mapper
+ );
+ }
+
+ private static ListeningExecutorService directExec()
+ {
+ return MoreExecutors.listeningDecorator(MoreExecutors.newDirectExecutorService());
+ }
+
+ private static PartialBundleAcquirer noOpAcquirer(ListeningExecutorService downloadExec)
+ {
+ return new PartialBundleAcquirer()
+ {
+ @Override
+ public Closeable acquire(String bundleName)
+ {
+ return () -> {};
+ }
+
+ @Override
+ public ListeningExecutorService getDownloadExec()
+ {
+ return downloadExec;
+ }
+ };
+ }
+
+ @Test
+ void testSyncMakeCursorHolderThrowsWhenSegmentNotFullyDownloaded() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_throws")) {
+ final PartialQueryableIndex index = opened.index();
+ Assertions.assertFalse(index.isFullyDownloaded());
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Fresh mapper has only the header on disk; no internal files have been downloaded. The sync path must
+ // refuse rather than trigger downloads on the calling thread.
+ Assertions.assertThrows(
+ DruidException.class,
+ () -> factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)
+ );
+ }
+ }
+
+ @Test
+ void testSyncMakeCursorHolderSucceedsAfterFullDownload() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_after_async")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Eagerly materialize the entire segment via the file mapper to simulate the eager-acquire path.
+ mapper.ensureFilesAvailable(mapper.getSegmentFileMetadata().getFiles().keySet());
+ Assertions.assertTrue(mapper.isFullyDownloaded(), "test precondition: every internal file should be on disk");
+
+ try (CursorHolder holder = factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)) {
+ Assertions.assertNotNull(holder);
+ }
+ }
+ }
+
+ @Test
+ void testAsyncMakeCursorHolderDefersDownloadUntilExecutorRuns() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ final CountDownLatch gate = new CountDownLatch(1);
+ final ExecutorService rawExec = Execs.singleThreaded("partial-cursor-defer-%d");
+ final ListeningExecutorService gatedExec = MoreExecutors.listeningDecorator(rawExec);
+ try (IndexAndMapper opened = openIndex(rangeReader, "async_defer")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(gatedExec)
+ );
+ rangeReader.resetCount();
+
+ // Hold the executor with a pre-queued blocker so the actual download task can't start.
+ // assign the blocker future to an (intentionally unused) local so errorprone's CheckReturnValue is satisfied
+ @SuppressWarnings("unused")
+ ListenableFuture<?> unused = gatedExec.submit(() -> {
+ try {
+ gate.await();
+ }
+ catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ });
+
+ final AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(CursorBuildSpec.FULL_SCAN);
+ Assertions.assertFalse(
+ asyncHolder.isReady(),
+ "async holder must not be ready while download executor is still blocked"
+ );
+ Assertions.assertEquals(0, rangeReader.getReadCount(), "no download should have started yet");
+
+ final CountDownLatch ready = new CountDownLatch(1);
+ asyncHolder.addReadyCallback(ready::countDown);
+
+ // Release the gate; download should now run.
+ gate.countDown();
+ Assertions.assertTrue(ready.await(5, TimeUnit.SECONDS), "ready callback must fire after executor processes task");
+ Assertions.assertTrue(asyncHolder.isReady());
+ Assertions.assertTrue(rangeReader.getReadCount() > 0, "download must have happened by the time holder is ready");
+
+ try (CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ }
+ }
+ finally {
+ gatedExec.shutdownNow();
+ rawExec.shutdownNow();
+ }
+ }
+
+ @Test
+ void testMatchedProjectionDownloadsOnlyRequestedColumns() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "proj_columns")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Query asks for group-by dim1 + sum(metric1), but NOT count. The projection has dim1 / _metric1_sum / _count;
+ // the projection match rewrites this to physical columns {dim1, _metric1_sum}. We should see those download,
+ // but NOT the _count file; proves column-level pruning, not just projection-level.
+ final CursorBuildSpec aggSpec = CursorBuildSpec.builder()
+ .setGroupingColumns(List.of("dim1"))
+ .setAggregators(List.of(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1")
+ ))
+ .setPhysicalColumns(Set.of("dim1", "metric1"))
+ .build();
+
+ // Async path triggers the column-level pre-download. directExec makes the future complete synchronously.
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(aggSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ final String projPrefix = PROJECTION_NAME + "/";
+ final String basePrefix = Projections.BASE_TABLE_PROJECTION_NAME + "/";
+ final Set<String> downloaded = mapper.getDownloadedFiles();
+
+ // Requested projection columns materialized.
+ Assertions.assertTrue(downloaded.contains(projPrefix + "dim1"), "expected projection dim1; got: " + downloaded);
+ Assertions.assertTrue(
+ downloaded.contains(projPrefix + "_metric1_sum"),
+ "expected projection _metric1_sum; got: " + downloaded
+ );
+ // The projection's _count file was NOT in the query; must not be downloaded
+ Assertions.assertFalse(
+ downloaded.contains(projPrefix + "_count"),
+ "expected projection _count NOT to be downloaded; got: " + downloaded
+ );
+ // Base __time and base metric1 are NOT touched: projection dim1 may pull base dim1 as its parent column
+ // (legitimate dependency), but unrelated base columns must stay untouched.
+ Assertions.assertFalse(
+ downloaded.contains(basePrefix + ColumnHolder.TIME_COLUMN_NAME),
+ "expected base __time NOT to be downloaded; got: " + downloaded
+ );
+ Assertions.assertFalse(
+ downloaded.contains(basePrefix + "metric1"),
+ "expected base metric1 NOT to be downloaded; got: " + downloaded
+ );
+ }
+ }
+ }
+
+ @Test
+ void testOpeningTimeOrderedProjectionCursorTriggersNoDownload() throws IOException
+ {
+ // A projection grouped by [__gran(HOUR), dim1] is time-ordered, so QueryableIndexCursorHolder reads the
+ // projection's (downloadable) time column when it builds the cursor (interval-checking offset + the projection's
+ // own time-boundary inspector). A GROUP BY dim1 query matches the projection by re-aggregating over __gran and
+ // does NOT list __time in physicalColumns. requiredColumns must nonetheless pre-fetch __time so that read happens
+ // during the async pre-fetch rather than as a lazy download when the cursor is opened on a processing thread.
+ final String projectionName = "hourly_dim1_metric1_sum";
+ final File timeOrderedSegmentDir = buildTimeOrderedProjectionSegment(projectionName);
+
+ final CountingRangeReader rangeReader = new CountingRangeReader(timeOrderedSegmentDir);
+ final File cacheDir = new File(perTestTempDir, "time_ordered_proj");
+ FileUtils.mkdirp(cacheDir);
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ TestHelper.makeJsonMapper(),
+ cacheDir,
+ IndexIO.V10_FILE_NAME,
+ Collections.emptyList()
+ );
+ try {
+ final PartialQueryableIndex index =
+ new PartialQueryableIndex(mapper.getSegmentFileMetadata(), mapper, COLUMN_CONFIG);
+
+ // Precondition: the matched projection's row selector really is time-ordered, so cursor build WOULD read its
+ // time column. Without this, the assertion below would be vacuously satisfied.
+ final QueryableIndex projectionIndex = index.getProjectionQueryableIndex(projectionName);
+ Assertions.assertNotNull(projectionIndex, "test setup: projection should be present");
+ Assertions.assertEquals(
+ Order.ASCENDING,
+ Cursors.getTimeOrdering(projectionIndex.getOrdering()),
+ "test setup: projection must be time-ordered for this test to guard anything"
+ );
+
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+
+ // ETERNITY interval so the offset is not clipped to a residual interval (a genuine reason to read __time);
+ // here the only __time read is the time-ordered offset/inspector during cursor build, which the pre-fetch
+ // must have already covered. physicalColumns deliberately omits __time.
+ final CursorBuildSpec aggSpec = CursorBuildSpec.builder()
+ .setInterval(Intervals.ETERNITY)
+ .setGroupingColumns(List.of("dim1"))
+ .setAggregators(List.of(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1")
+ ))
+ .setPhysicalColumns(Set.of("dim1", "metric1"))
+ .build();
+
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(aggSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+
+ // After the async pre-fetch resolves, the projection's required columns AND __time are on disk, even though
+ // __time was not declared in physicalColumns.
+ final String projPrefix = projectionName + "/";
+ final Set<String> beforeOpen = mapper.getDownloadedFiles();
+ Assertions.assertTrue(
+ beforeOpen.contains(projPrefix + "_metric1_sum"),
+ "test setup: projection path should have run and pre-fetched _metric1_sum; got: " + beforeOpen
+ );
+ // The projection's time column is stored under its granularity grouping name (__gran); it is exposed as
+ // __time but the on-disk smoosh file (and thus the downloaded-files entry) is __gran.
+ Assertions.assertTrue(
+ beforeOpen.contains(projPrefix + "__gran"),
+ "requiredColumns must pre-fetch the projection's time column for a time-ordered cursor; got: " + beforeOpen
+ );
+
+ // Opening the cursor reads __time (interval-checking offset + the projection's time-boundary inspector). It
+ // was pre-fetched above, so this must not trigger any further (lazy) download on the processing thread.
+ Assertions.assertNotNull(holder.asCursor());
+
+ final Set<String> afterOpen = mapper.getDownloadedFiles();
+ Assertions.assertEquals(
+ beforeOpen,
+ afterOpen,
+ "opening a projection-matched cursor must not trigger a lazy download; newly downloaded: "
+ + Sets.difference(afterOpen, beforeOpen)
+ );
+ }
+ }
+ finally {
+ mapper.close();
+ }
+ }
+
+ @Test
+ void testOpeningTimeOrderedBaseCursorTriggersNoDownload() throws IOException
+ {
+ // The base table is time-ordered, so QueryableIndexCursorHolder reads __time when building the cursor (for its
+ // interval-checking offset) even though a raw scan does not list __time in physicalColumns. requiredColumns must
+ // pre-fetch __time so that read does not become a lazy download on the processing thread at cursor-open time.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "base_open_no_download")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ Assertions.assertEquals(
+ Order.ASCENDING,
+ Cursors.getTimeOrdering(index.getOrdering()),
+ "test setup: base table must be time-ordered"
+ );
+
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+
+ // Raw scan (no grouping/aggregators) so the aggregate projection is not matched and we exercise the base path.
+ // __time is deliberately omitted from physicalColumns.
+ final CursorBuildSpec scanSpec = CursorBuildSpec.builder()
+ .setInterval(Intervals.ETERNITY)
+ .setPhysicalColumns(Set.of("dim1"))
+ .build();
+
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(scanSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+
+ final String basePrefix = Projections.BASE_TABLE_PROJECTION_NAME + "/";
+ final Set<String> beforeOpen = mapper.getDownloadedFiles();
+ // base path ran (dim1 pre-fetched) and __time was pre-fetched despite not being in physicalColumns
+ Assertions.assertTrue(
+ beforeOpen.contains(basePrefix + "dim1"),
+ "test setup: base path should have pre-fetched dim1; got: " + beforeOpen
+ );
+ Assertions.assertTrue(
+ beforeOpen.contains(basePrefix + ColumnHolder.TIME_COLUMN_NAME),
+ "requiredColumns must pre-fetch __time for a time-ordered base cursor; got: " + beforeOpen
+ );
+
+ Assertions.assertNotNull(holder.asCursor());
+
+ final Set<String> afterOpen = mapper.getDownloadedFiles();
+ Assertions.assertEquals(
+ beforeOpen,
+ afterOpen,
+ "opening a time-ordered base cursor must not trigger a lazy download; newly downloaded: "
+ + Sets.difference(afterOpen, beforeOpen)
+ );
+ }
+ }
+ }
+
+ @Test
+ void testFilteredWrapperOverPartialUsesAsyncPath() throws IOException
+ {
+ // A FilteredCursorFactory wrapping a not-fully-downloaded partial cursor factory must route makeCursorHolderAsync
+ // through the delegate's async (download-on-demand) path. If FilteredCursorFactory fell back to the CursorFactory
+ // default makeCursorHolderAsync (which calls sync makeCursorHolder), the partial delegate would throw because the
+ // segment isn't fully downloaded.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "filtered_wrapper_async")) {
+ final PartialQueryableIndex index = opened.index();
+ Assertions.assertFalse(index.isFullyDownloaded(), "test setup: segment must not be fully downloaded");
+
+ final PartialQueryableIndexCursorFactory partial = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+ final CursorFactory filtered = new FilteredCursorFactory(
+ partial,
+ new EqualityFilter("dim1", ColumnType.STRING, "a", null)
+ );
+
+ // directExec resolves the async download inline; release() returns the built holder. asCursor() exercises the
+ // cursor over the on-demand-loaded columns. Would throw here if the wrapper used the sync default.
+ try (AsyncCursorHolder asyncHolder = filtered.makeCursorHolderAsync(CursorBuildSpec.FULL_SCAN);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ Assertions.assertNotNull(holder.asCursor(), "async path should build a usable cursor without a sync download");
+ }
+ }
+ }
+
+ private File buildTimeOrderedProjectionSegment(String projectionName)
+ {
+ final List<AggregateProjectionSpec> projections = Collections.singletonList(
+ AggregateProjectionSpec.builder(projectionName)
+ .virtualColumns(Granularities.toVirtualColumn(Granularities.HOUR, "__gran"))
+ .groupingColumns(
+ new LongDimensionSchema("__gran"),
+ new StringDimensionSchema("dim1")
+ )
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+ final File tmpDir = new File(perTestTempDir, "build_time_ordered_" + ThreadLocalRandom.current().nextInt(Integer.MAX_VALUE));
+ return IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmpDir)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(projections)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder().withMetadataCompression(CompressionStrategy.NONE).build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+ }
+
+ @Test
+ void testRowSignatureAndCapabilitiesDelegatedToUnderlyingFactory() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "delegate")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // delegating to QueryableIndexCursorFactory which returns the index's row signature
+ final RowSignature signature = factory.getRowSignature();
+ Assertions.assertNotNull(signature);
+ Assertions.assertTrue(signature.size() > 0);
+
+ Assertions.assertNotNull(factory.getColumnCapabilities("dim1"));
+ }
+ }
+
+ @Test
+ void testAsyncCloseBeforeDownloadCompletesClosesCursorHolderOnce() throws Exception
+ {
+ // Producer-doesn't-leak invariant: if the wrapper is closed before the producer's set(holder) fires, the producer
+ // is responsible for closing the holder. Use the real (single-threaded) executor with a gate that pauses the
+ // download task before it builds the cursor, then close the wrapper, then release the gate.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ final AtomicReference<RuntimeException> bgErr = new AtomicReference<>();
+ final CountDownLatch gate = new CountDownLatch(1);
+ final ExecutorService rawExec = Execs.singleThreaded("partial-cursor-close-%d");
+ final ListeningExecutorService gatedExec = MoreExecutors.listeningDecorator(rawExec);
+ try (IndexAndMapper opened = openIndex(rangeReader, "close_before_ready")) {
+ final PartialQueryableIndex index = opened.index();
+ // Queue the gate first so the download task can't start until we release it.
+ // assign the blocker future to an (intentionally unused) local so errorprone's CheckReturnValue is satisfied
+ @SuppressWarnings("unused")
+ ListenableFuture<?> unused = gatedExec.submit(() -> {
+ try {
+ gate.await();
+ }
+ catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ });
Review Comment:
## CodeQL / Unread local variable
Variable 'ListenableFuture<?> unused' is never read.
[Show more details](https://github.com/apache/druid/security/code-scanning/11288)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3333814034
##########
server/src/main/java/org/apache/druid/segment/loading/PartialSegmentMetadataCacheEntry.java:
##########
@@ -236,15 +279,8 @@
// clear the future so the next caller gets a fresh attempt
mountFuture.set(null);
ours.setException(t);
- if (t instanceof IOException) {
- throw (IOException) t;
- }
- if (t instanceof RuntimeException) {
- throw (RuntimeException) t;
- }
- if (t instanceof Error) {
- throw (Error) t;
- }
+ Throwables.propagateIfInstanceOf(t, IOException.class);
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [Throwables.propagateIfInstanceOf](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11273)
##########
server/src/main/java/org/apache/druid/segment/loading/PartialSegmentMetadataCacheEntry.java:
##########
@@ -236,15 +279,8 @@
// clear the future so the next caller gets a fresh attempt
mountFuture.set(null);
ours.setException(t);
- if (t instanceof IOException) {
- throw (IOException) t;
- }
- if (t instanceof RuntimeException) {
- throw (RuntimeException) t;
- }
- if (t instanceof Error) {
- throw (Error) t;
- }
+ Throwables.propagateIfInstanceOf(t, IOException.class);
+ Throwables.propagateIfPossible(t);
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [Throwables.propagateIfPossible](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11274)
##########
server/src/main/java/org/apache/druid/segment/loading/PartialSegmentMetadataCacheEntry.java:
##########
@@ -280,87 +316,129 @@
// blocked on it. adjustReservation also runs outside entryLock: StorageLocation.release goes
// writeLock -> entryLock (via release -> unmount), so entryLock -> writeLock here would be a deadlock-prone
// lock-order inversion.
- entryLock.lock();
try {
- if (location != null && fileMapper != null) {
- if (!location.equals(mountLocation)) {
- throw DruidException.defensive(
- "Already mounted[%s] in location[%s] which differs from requested[%s]",
- id,
- location.getPath(),
- mountLocation.getPath()
- );
+ entryLock.lock();
+ try {
+ if (location != null && fileMapper != null) {
+ if (!location.equals(mountLocation)) {
+ throw DruidException.defensive(
+ "Already mounted[%s] in location[%s] which differs from requested[%s]",
+ id,
+ location.getPath(),
+ mountLocation.getPath()
+ );
+ }
+ return;
}
- return;
}
- }
- finally {
- entryLock.unlock();
- }
+ finally {
+ entryLock.unlock();
+ }
- final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
- rangeReader,
- jsonMapper,
- localCacheDir,
- targetFilename,
- externalFilenames
- );
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ jsonMapper,
+ localCacheDir,
+ targetFilename,
+ externalFilenames
+ );
- final long sizeToAdjust;
- try {
- final long actualSize = mapper.getOnDiskHeaderSize();
- if (actualSize > reservationEstimate) {
- throw DruidException.forPersona(DruidException.Persona.OPERATOR)
- .ofCategory(DruidException.Category.RUNTIME_FAILURE)
- .build(
- "Partial segment metadata for [%s] is [%d] bytes on disk, exceeding the "
- + "configured reservation estimate of [%d] bytes. Increase "
- + "druid.segmentCache.virtualStorageMetadataReservationEstimate.",
- segmentId,
- actualSize,
- reservationEstimate
- );
+ final long sizeToAdjust;
+ try {
+ final long actualSize = mapper.getOnDiskHeaderSize();
+ if (actualSize > reservationEstimate) {
+ throw DruidException.forPersona(DruidException.Persona.OPERATOR)
+ .ofCategory(DruidException.Category.RUNTIME_FAILURE)
+ .build(
+ "Partial segment metadata for [%s] is [%d] bytes on disk, exceeding the "
+ + "configured reservation estimate of [%d] bytes. Increase "
+ + "druid.segmentCache.virtualStorageMetadataReservationEstimate.",
+ segmentId,
+ actualSize,
+ reservationEstimate
+ );
+ }
+ sizeToAdjust = actualSize < reservationEstimate ? actualSize : -1;
+
+ entryLock.lock();
+ try {
+ location = mountLocation;
+ fileMapper = mapper;
+ // Install (or re-install, after a previous mount/unmount cycle terminated the prior Phaser) the
+ // reference-counted gate over cleanup. Future acquireMetadataReference() / unmount() calls operate on this
+ // instance.
+ references.set(new ReferenceCountingCloseableObject<Closeable>(this::doActualUnmount) {});
+ }
+ finally {
+ entryLock.unlock();
+ }
+ }
+ catch (Throwable t) {
+ // mount failed; close mmaps and delete the on-disk header files so a retry starts clean. Mirrors the eager
+ // SegmentCacheEntry behavior: simpler to redo a small header range-read than to reason about whatever partial
+ // on-disk state the failure left. Crash-mid-mount across JVM restarts is still handled by the mapper's own
+ // corruption recovery when bootstrap runs at next startup; this path covers the in-process retry case.
+ try {
+ mapper.close();
+ }
+ catch (Throwable closeError) {
+ t.addSuppressed(closeError);
+ }
+ try {
+ deleteHeaderFiles();
+ }
+ catch (Throwable deleteError) {
+ t.addSuppressed(deleteError);
+ }
+ throw t;
}
- sizeToAdjust = actualSize < reservationEstimate ? actualSize : -1;
- entryLock.lock();
+ // Only shrink the reservation if the entry is still registered with the location. If we lost the reservation
+ // mid-mount (concurrent canceler / drop), adjustReservation would throw; defer to the post-mount check in
+ // mount() to roll back cleanly instead.
+ if (sizeToAdjust >= 0 && (mountLocation.isReserved(id) || mountLocation.isWeakReserved(id))) {
+ mountLocation.adjustReservation(id, sizeToAdjust);
+ }
+
+ // Restore any bundles whose container files survived on disk for this segment. No-op on the fresh-acquire path
+ // (the partialDir was just created and contains no container files yet). The bundle restore needs metadata's
+ // file mapper, which is now installed, so this runs after the commit above. On any failure, fire our own
+ // deferred-cleanup gate via unmount() to undo the file-mapper install + delete header files; synchronous since
+ // we just installed the gate and no external caller has had a chance to acquire a reference yet. The location
+ // reservation release stays the caller's responsibility (matches mount's overall contract).
try {
- location = mountLocation;
- fileMapper = mapper;
- // Install (or re-install, after a previous mount/unmount cycle terminated the prior Phaser) the
- // reference-counted gate over cleanup. Future acquireReference() / unmount() calls operate on this instance.
- references.set(new ReferenceCountingCloseableObject<Closeable>(this::doActualUnmount) {});
+ PartialSegmentCacheBootstrap.restoreBundlesFromDisk(this, mountLocation);
}
- finally {
- entryLock.unlock();
+ catch (Throwable t) {
+ try {
+ unmount();
+ }
+ catch (Throwable cleanupError) {
+ t.addSuppressed(cleanupError);
+ }
+ Throwables.propagateIfInstanceOf(t, IOException.class);
+ Throwables.propagateIfPossible(t);
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [Throwables.propagateIfPossible](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11276)
##########
server/src/main/java/org/apache/druid/segment/loading/PartialSegmentMetadataCacheEntry.java:
##########
@@ -280,87 +316,129 @@
// blocked on it. adjustReservation also runs outside entryLock: StorageLocation.release goes
// writeLock -> entryLock (via release -> unmount), so entryLock -> writeLock here would be a deadlock-prone
// lock-order inversion.
- entryLock.lock();
try {
- if (location != null && fileMapper != null) {
- if (!location.equals(mountLocation)) {
- throw DruidException.defensive(
- "Already mounted[%s] in location[%s] which differs from requested[%s]",
- id,
- location.getPath(),
- mountLocation.getPath()
- );
+ entryLock.lock();
+ try {
+ if (location != null && fileMapper != null) {
+ if (!location.equals(mountLocation)) {
+ throw DruidException.defensive(
+ "Already mounted[%s] in location[%s] which differs from requested[%s]",
+ id,
+ location.getPath(),
+ mountLocation.getPath()
+ );
+ }
+ return;
}
- return;
}
- }
- finally {
- entryLock.unlock();
- }
+ finally {
+ entryLock.unlock();
+ }
- final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
- rangeReader,
- jsonMapper,
- localCacheDir,
- targetFilename,
- externalFilenames
- );
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ jsonMapper,
+ localCacheDir,
+ targetFilename,
+ externalFilenames
+ );
- final long sizeToAdjust;
- try {
- final long actualSize = mapper.getOnDiskHeaderSize();
- if (actualSize > reservationEstimate) {
- throw DruidException.forPersona(DruidException.Persona.OPERATOR)
- .ofCategory(DruidException.Category.RUNTIME_FAILURE)
- .build(
- "Partial segment metadata for [%s] is [%d] bytes on disk, exceeding the "
- + "configured reservation estimate of [%d] bytes. Increase "
- + "druid.segmentCache.virtualStorageMetadataReservationEstimate.",
- segmentId,
- actualSize,
- reservationEstimate
- );
+ final long sizeToAdjust;
+ try {
+ final long actualSize = mapper.getOnDiskHeaderSize();
+ if (actualSize > reservationEstimate) {
+ throw DruidException.forPersona(DruidException.Persona.OPERATOR)
+ .ofCategory(DruidException.Category.RUNTIME_FAILURE)
+ .build(
+ "Partial segment metadata for [%s] is [%d] bytes on disk, exceeding the "
+ + "configured reservation estimate of [%d] bytes. Increase "
+ + "druid.segmentCache.virtualStorageMetadataReservationEstimate.",
+ segmentId,
+ actualSize,
+ reservationEstimate
+ );
+ }
+ sizeToAdjust = actualSize < reservationEstimate ? actualSize : -1;
+
+ entryLock.lock();
+ try {
+ location = mountLocation;
+ fileMapper = mapper;
+ // Install (or re-install, after a previous mount/unmount cycle terminated the prior Phaser) the
+ // reference-counted gate over cleanup. Future acquireMetadataReference() / unmount() calls operate on this
+ // instance.
+ references.set(new ReferenceCountingCloseableObject<Closeable>(this::doActualUnmount) {});
+ }
+ finally {
+ entryLock.unlock();
+ }
+ }
+ catch (Throwable t) {
+ // mount failed; close mmaps and delete the on-disk header files so a retry starts clean. Mirrors the eager
+ // SegmentCacheEntry behavior: simpler to redo a small header range-read than to reason about whatever partial
+ // on-disk state the failure left. Crash-mid-mount across JVM restarts is still handled by the mapper's own
+ // corruption recovery when bootstrap runs at next startup; this path covers the in-process retry case.
+ try {
+ mapper.close();
+ }
+ catch (Throwable closeError) {
+ t.addSuppressed(closeError);
+ }
+ try {
+ deleteHeaderFiles();
+ }
+ catch (Throwable deleteError) {
+ t.addSuppressed(deleteError);
+ }
+ throw t;
}
- sizeToAdjust = actualSize < reservationEstimate ? actualSize : -1;
- entryLock.lock();
+ // Only shrink the reservation if the entry is still registered with the location. If we lost the reservation
+ // mid-mount (concurrent canceler / drop), adjustReservation would throw; defer to the post-mount check in
+ // mount() to roll back cleanly instead.
+ if (sizeToAdjust >= 0 && (mountLocation.isReserved(id) || mountLocation.isWeakReserved(id))) {
+ mountLocation.adjustReservation(id, sizeToAdjust);
+ }
+
+ // Restore any bundles whose container files survived on disk for this segment. No-op on the fresh-acquire path
+ // (the partialDir was just created and contains no container files yet). The bundle restore needs metadata's
+ // file mapper, which is now installed, so this runs after the commit above. On any failure, fire our own
+ // deferred-cleanup gate via unmount() to undo the file-mapper install + delete header files; synchronous since
+ // we just installed the gate and no external caller has had a chance to acquire a reference yet. The location
+ // reservation release stays the caller's responsibility (matches mount's overall contract).
try {
- location = mountLocation;
- fileMapper = mapper;
- // Install (or re-install, after a previous mount/unmount cycle terminated the prior Phaser) the
- // reference-counted gate over cleanup. Future acquireReference() / unmount() calls operate on this instance.
- references.set(new ReferenceCountingCloseableObject<Closeable>(this::doActualUnmount) {});
+ PartialSegmentCacheBootstrap.restoreBundlesFromDisk(this, mountLocation);
}
- finally {
- entryLock.unlock();
+ catch (Throwable t) {
+ try {
+ unmount();
+ }
+ catch (Throwable cleanupError) {
+ t.addSuppressed(cleanupError);
+ }
+ Throwables.propagateIfInstanceOf(t, IOException.class);
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [Throwables.propagateIfInstanceOf](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11275)
##########
server/src/test/java/org/apache/druid/segment/loading/SegmentLocalCacheManagerPartialAcquireTest.java:
##########
@@ -0,0 +1,527 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.loading;
+
+import com.fasterxml.jackson.databind.InjectableValues;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.jsontype.NamedType;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.ListBasedInputRow;
+import org.apache.druid.data.input.impl.AggregateProjectionSpec;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.LongDimensionSchema;
+import org.apache.druid.data.input.impl.StringDimensionSchema;
+import org.apache.druid.guice.LocalDataStorageDruidModule;
+import org.apache.druid.jackson.SegmentizerModule;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import org.apache.druid.query.expression.TestExprMacroTable;
+import org.apache.druid.segment.CursorBuildSpec;
+import org.apache.druid.segment.CursorFactory;
+import org.apache.druid.segment.CursorHolder;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.IndexIO;
+import org.apache.druid.segment.IndexSpec;
+import org.apache.druid.segment.PartialQueryableIndexSegment;
+import org.apache.druid.segment.Segment;
+import org.apache.druid.segment.SegmentLazyLoadFailCallback;
+import org.apache.druid.segment.TestHelper;
+import org.apache.druid.segment.TimeBoundaryInspector;
+import org.apache.druid.segment.V10TimeBoundaryInspector;
+import org.apache.druid.segment.column.ColumnConfig;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.CompressionStrategy;
+import org.apache.druid.segment.file.PartialSegmentFileMapperV10;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.server.metrics.NoopServiceEmitter;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.partition.NoneShardSpec;
+import org.apache.druid.utils.CloseableUtils;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+class SegmentLocalCacheManagerPartialAcquireTest
+{
+ private static final SegmentId SEGMENT_ID = SegmentId.of("test", Intervals.of("2025/2026"), "v1", 0);
+ private static final DateTime TIME = DateTimes.of("2025-01-01");
+ private static final String AGG_BUNDLE = "dim1_metric1_sum";
+
+ private static final RowSignature ROW_SIGNATURE = RowSignature.builder()
+ .add("dim1", ColumnType.STRING)
+ .add("metric1", ColumnType.LONG)
+ .build();
+
+ private static final List<AggregateProjectionSpec> PROJECTIONS = Collections.singletonList(
+ AggregateProjectionSpec.builder(AGG_BUNDLE)
+ .groupingColumns(new StringDimensionSchema("dim1"))
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+
+ private static final List<InputRow> ROWS = Arrays.asList(
+ new ListBasedInputRow(ROW_SIGNATURE, TIME, ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 1L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(1), ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 2L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(2), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 3L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(3), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 4L))
+ );
+
+ @TempDir
+ static File SHARED_TEMP_DIR;
+
+ private static File DEEP_STORAGE_DIR;
+
+ @TempDir
+ File perTestTempDir;
+
+ private ObjectMapper jsonMapper;
+ private File cacheRoot;
+ private SegmentLocalCacheManager manager;
+ private DataSegment partialSegment;
+
+ @BeforeAll
+ static void buildSegment()
+ {
+ final File tmp = new File(SHARED_TEMP_DIR, "build_" + ThreadLocalRandom.current().nextInt());
+ DEEP_STORAGE_DIR = IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmp)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(PROJECTIONS)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder()
+ .withMetadataCompression(CompressionStrategy.NONE)
+ .build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+ EmittingLogger.registerEmitter(new NoopServiceEmitter());
+ }
+
+ @BeforeEach
+ void setup() throws IOException
+ {
+ jsonMapper = TestHelper.makeJsonMapper();
+ jsonMapper.registerSubtypes(new NamedType(LocalLoadSpec.class, "local"));
+ jsonMapper.registerModule(new SegmentizerModule());
+ jsonMapper.registerModules(new LocalDataStorageDruidModule().getJacksonModules());
+ jsonMapper.setInjectableValues(
+ new InjectableValues.Std()
+ .addValue(LocalDataSegmentPuller.class, new LocalDataSegmentPuller())
+ .addValue(IndexIO.class, TestHelper.getTestIndexIO(jsonMapper, ColumnConfig.DEFAULT))
+ .addValue(ObjectMapper.class, jsonMapper)
+ .addValue(DataSegment.PruneSpecsHolder.class, DataSegment.PruneSpecsHolder.DEFAULT)
+ .addValue(ExprMacroTable.class, TestExprMacroTable.INSTANCE)
+ );
+
+ cacheRoot = new File(perTestTempDir, "cache_" + ThreadLocalRandom.current().nextInt(Integer.MAX_VALUE));
+ FileUtils.mkdirp(cacheRoot);
+
+ final StorageLocationConfig locConfig = new StorageLocationConfig(cacheRoot, 1024L * 1024L * 1024L, null);
+ final SegmentLoaderConfig loaderConfig = new SegmentLoaderConfig()
+ .setLocations(List.of(locConfig))
+ .setVirtualStorage(true, false);
+ final List<StorageLocation> storageLocations = loaderConfig.toStorageLocations();
+ manager = new SegmentLocalCacheManager(
+ storageLocations,
+ loaderConfig,
+ new LeastBytesUsedStorageLocationSelectorStrategy(storageLocations),
+ TestHelper.getTestIndexIO(jsonMapper, ColumnConfig.DEFAULT),
+ jsonMapper
+ )
+ {
+ // Tripwire: the partial-aware acquire paths should be self-contained; they must not fall through to the
+ // eager-extract acquireCachedSegment API (which is the non-partial branch of acquireCachedSegment-style lookups).
+ @Override
+ public Optional<Segment> acquireCachedSegment(SegmentId segmentId)
+ {
+ Assertions.fail("should not fallback to acquireCachedSegment");
+ return super.acquireCachedSegment(segmentId);
+ }
+ };
+
+ // DataSegment with a LocalLoadSpec pointing at the deep storage directory (unzipped V10 layout).
+ partialSegment = DataSegment.builder()
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [DataSegment.builder](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11277)
##########
server/src/test/java/org/apache/druid/segment/loading/SegmentLocalCacheManagerPartialDropTest.java:
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.loading;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.ListBasedInputRow;
+import org.apache.druid.data.input.impl.AggregateProjectionSpec;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.LongDimensionSchema;
+import org.apache.druid.data.input.impl.StringDimensionSchema;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.IndexIO;
+import org.apache.druid.segment.IndexSpec;
+import org.apache.druid.segment.TestHelper;
+import org.apache.druid.segment.column.ColumnConfig;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.CompressionStrategy;
+import org.apache.druid.segment.file.PartialSegmentFileMapperV10;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.partition.NoneShardSpec;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Manager-level tests for {@code drop} / {@code removeInfoFile} on partial segments. Partial entries (metadata +
+ * bundles) are weak cache entries; their lifecycle is owned by {@link StorageLocation}'s SIEVE + hold-phaser
+ * machinery. Coordinator {@code drop} is effectively a no-op for the partial branch — SIEVE drives the actual
+ * eviction once holds drop and capacity is needed. {@code removeInfoFile} still wires info-file deletion via the
+ * metadata entry's {@link PartialSegmentMetadataCacheEntry#setOnUnmount} hook so the info entry survives until the
+ * (now SIEVE-driven) cleanup actually fires.
+ * <p>
+ * These tests use {@link StorageLocation#setAreWeakEntriesEphemeral} so hold-release triggers immediate eviction,
+ * a deterministic substitute for SIEVE's capacity-pressure-driven reclaim.
+ */
+class SegmentLocalCacheManagerPartialDropTest extends InitializedNullHandlingTest
+{
+ private static final ObjectMapper JSON_MAPPER = TestHelper.makeJsonMapper();
+ private static final SegmentId SEGMENT_ID = SegmentId.of("test", Intervals.of("2025/2026"), "v1", 0);
+ private static final long ESTIMATE = 16 * 1024 * 1024L;
+ private static final String AGG_BUNDLE = "dim1_metric1_sum";
+
+ private static final DateTime TIME = DateTimes.of("2025-01-01");
+ private static final RowSignature ROW_SIGNATURE = RowSignature.builder()
+ .add("dim1", ColumnType.STRING)
+ .add("metric1", ColumnType.LONG)
+ .build();
+
+ private static final List<AggregateProjectionSpec> PROJECTIONS = Collections.singletonList(
+ AggregateProjectionSpec.builder(AGG_BUNDLE)
+ .groupingColumns(new StringDimensionSchema("dim1"))
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+
+ private static final List<InputRow> ROWS = Arrays.asList(
+ new ListBasedInputRow(ROW_SIGNATURE, TIME, ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 1L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(1), ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 2L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(2), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 3L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(3), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 4L))
+ );
+
+ @TempDir
+ static File sharedTempDir;
+
+ private static File deepStorageDir;
+
+ @TempDir
+ File perTestTempDir;
+
+ private File cacheDir;
+ private File infoDir;
+ private SegmentLocalCacheManager manager;
+ private StorageLocation location;
+ private DataSegment dataSegment;
+
+ @BeforeAll
+ static void buildSegment()
+ {
+ final File tmp = new File(sharedTempDir, "build_" + ThreadLocalRandom.current().nextInt());
+ deepStorageDir = IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmp)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(PROJECTIONS)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder().withMetadataCompression(CompressionStrategy.NONE).build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+ }
+
+ @BeforeEach
+ void setup() throws IOException
+ {
+ cacheDir = new File(perTestTempDir, "cache_" + ThreadLocalRandom.current().nextInt(Integer.MAX_VALUE));
+ FileUtils.mkdirp(cacheDir);
+ // SegmentLocalCacheManager defaults the info dir to <firstLocation>/info_dir when not configured.
+ infoDir = new File(cacheDir, "info_dir");
+ FileUtils.mkdirp(infoDir);
+
+ final StorageLocationConfig locConfig = new StorageLocationConfig(cacheDir, ESTIMATE * 16, null);
+ final SegmentLoaderConfig loaderConfig = new SegmentLoaderConfig().setLocations(List.of(locConfig));
+ final List<StorageLocation> locations = loaderConfig.toStorageLocations();
+ manager = new SegmentLocalCacheManager(
+ locations,
+ loaderConfig,
+ new LeastBytesUsedStorageLocationSelectorStrategy(locations),
+ TestHelper.getTestIndexIO(JSON_MAPPER, ColumnConfig.DEFAULT),
+ JSON_MAPPER
+ );
+ location = manager.getLocations().get(0);
+ // Drive eviction off hold-release so tests don't need to provoke SIEVE via capacity pressure.
+ location.setAreWeakEntriesEphemeral(true);
+ dataSegment = DataSegment.builder()
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [DataSegment.builder](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11278)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415467548
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
Review Comment:
this is kind of fragile and magic I agree, I'll look into tightening this up, though maybe as a follow-up. Currently this is just baked into how v10 segments are written, we start a bundle every time we write a projection, but it would be nicer to like look this up in some way or push it into the filemapper or.. something, idk still thinking on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415395921
##########
multi-stage-query/src/main/java/org/apache/druid/msq/input/RegularLoadableSegment.java:
##########
@@ -140,7 +140,7 @@ public synchronized Optional<Segment> acquireIfCached()
throw DruidException.defensive("Segment with descriptor[%s] is already acquired", descriptor);
}
- final Optional<Segment> cachedSegment = segmentManager.acquireCachedSegment(segmentId);
+ final Optional<Segment> cachedSegment = segmentManager.acquireCachedPartialSegment(segmentId);
Review Comment:
this now accepts the `AcquireMode` enum argument to pass through, and the engine decides what mode it wants when creating the `ReadableInputQueue` (see `BaseLeafStageProcessor.makeReadableInputQueue`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415512778
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle BEFORE submitting downloads. This sparse-allocates the bundle's container files
+ // and reserves the disk-usage accounting at bundle granularity. The returned Closeable releases the hold when
+ // the cursor closes, allowing the cache to later evict the bundle if needed.
+ // PartialBundleAcquirer.acquire() throws DruidException on failure.
+ final Closeable bundleHold = bundleAcquirer.acquire(bundleName);
+
+ try {
+ // Submit one materialization task per column so a multi-threaded download executor can fan them out
+ final ListeningExecutorService downloadExec = bundleAcquirer.getDownloadExec();
+ final List<ListenableFuture<?>> columnDownloads = new ArrayList<>(requiredColumns.size());
+ for (String column : requiredColumns) {
+ columnDownloads.add(downloadExec.submit(() -> {
+ rowSelector.getColumnHolder(column);
+ return null;
+ }));
+ }
+
+ // No canceler: downloads are allowed to complete even if the awaiter cancels. Once they complete and the inner
Review Comment:
added cancel wiring, i didn't do it at first since was a bit complicated for book-keeping, but added a thing to handle it so it should be good now (the _actual_ download parts in the mapper still don't interrupt so an in flight pull will not stop since there is even more complicated cleanup that would need to be done to do that), but the at least all of the parts leading up to that can bail out now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415453574
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
Review Comment:
I have left this as is for now because this path gets actually used when a partial segment is used in 'full load' mode (e.g. by the native engine instead of Dart/MSQ). These guards are mainly to ensure that the caller is using the correct `AcquireMode` to hold the segment during processing.
The default implementation is probably not very useful though and should maybe be an exception, I'll look into changing this (either in this PR or in a follow-up since I still need to wire up unnest and join cursor factories to have async implementations so could do it then too).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415527005
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.asyncresource.AsyncResource;
+import org.apache.druid.common.asyncresource.AsyncResources;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle before submitting downloads. Released on the cancel/failure paths (requestRelease)
+ // or handed to the produced cursor holder on success (releaser), exactly once, and never while an in-flight column
+ // download is mid-mapFile(). See BundleHoldRelease.
+ final BundleHoldRelease holdRelease = new BundleHoldRelease(bundleAcquirer.acquire(bundleName));
+
+ try {
+ // submit one materialization task per column so a multi-threaded download executor can fan them out
+ final List<AsyncResource<String>> columnDownloads = new ArrayList<>(requiredColumns.size());
+ for (String column : requiredColumns) {
+ columnDownloads.add(submitColumnDownload(rowSelector, column, holdRelease));
+ }
+ final AsyncResource<List<String>> downloaded = AsyncResources.collect(columnDownloads);
+
+ // Canceler runs if the awaiter closes this holder before it's ready (e.g. query cancel/timeout). Close the
+ // collected resource to cancel every column download that hasn't begun its deep-storage read yet (queued tasks
+ // are skipped; tasks parked on the download executor's permit are interrupted out of the interruptible wait
+ // before doing any I/O), then request the hold release through the handshake. Once the holder is produced and
+ // handed to set(), ownership transfers to the awaiter, which drains it via close() (cancel) or release()
+ // (success); the once-guard keeps the hold release safe across all of these paths.
+ final AsyncCursorHolder asyncHolder = new AsyncCursorHolder(() -> {
+ CloseableUtils.closeAndSuppressExceptions(downloaded, ignored -> {});
+ holdRelease.requestRelease();
+ });
+ downloaded.addReadyCallback(() -> {
+ final CursorHolder holder;
+ try {
+ downloaded.get(); // surfaces any column-download failure (or cancellation) as the cause
+ final CursorHolder inner = delegate.makeCursorHolderForProjection(spec, matched);
+ // The produced holder takes ownership of both inner and the bundle hold; from here, closing it releases both
+ holder = wrapWithBundleRelease(inner, holdRelease.releaser());
+ }
+ catch (Throwable t) {
+ // A column download failed or was canceled, this branch can fire while a sibling download is still
+ // mid-mapFile(), so the hold release must go through the handshake rather than dropping it here directly.
+ holdRelease.requestRelease();
+ asyncHolder.setException(t);
+ return;
+ }
+ if (!asyncHolder.set(holder)) {
+ // wrapper was closed (awaiter canceled) while we were producing the holder; close it ourselves so the
+ // holder, its inner, and the bundle hold don't leak.
+ holder.close();
+ }
+ });
+ return asyncHolder;
+ }
+ catch (Throwable t) {
+ // Failure between acquire and wiring up the downloads (submitDownload shut-down rejection, etc.). Ownership of
+ // the bundle hold hasn't transferred to the holder yet, so release it here.
+ throw CloseableUtils.closeAndWrapInCatch(t, holdRelease.releaser());
Review Comment:
this was a mistake actually, good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3438128234
##########
server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java:
##########
@@ -105,6 +105,15 @@ public class SegmentLoaderConfig
@JsonProperty("virtualStorageMetadataReservationEstimate")
private long virtualStorageMetadataReservationEstimate = 16L * 1024L * 1024L;
+ /**
+ * When true, partial-eligible V10 segments are mounted via the partial machinery and
+ * {@link SegmentCacheManager#acquireSegment} with {@link AcquireMode#PARTIAL} returns a metadata-anchored segment
+ * whose columns are downloaded on demand. When false (the default), {@link AcquireMode#PARTIAL} falls back to
+ * {@link AcquireMode#FULL} so the entire segment is downloaded up front (matching pre-partial-download behavior).
+ */
+ @JsonProperty("virtualStoragePartialDownloadsEnabled")
+ private boolean virtualStoragePartialDownloadsEnabled = false;
Review Comment:
IMO it would be better to have this always enabled at the VSF layer, and to have the toggle be in the query context. There can be something like `QueryContexts#getAcquireMode`. (Something like `usePartialDownloads: true/false`). I like this approach because it makes testing much easier, and also it makes the code in the VSF layer easier to follow because it gets rid of conditionals. There would also be no reason to get rid of partially-downloaded directories during bootstrapping, because they're always considered OK.
Default should be `usePartialDownloads: false` for now.
##########
multi-stage-query/src/main/java/org/apache/druid/msq/querykit/ReadableInputQueue.java:
##########
@@ -288,7 +298,7 @@ private ListenableFuture<ReadableInput> loadNextSegment()
return null;
}
- final AcquireSegmentAction acquireSegmentAction = nextLoadableSegment.acquire();
+ final AcquireSegmentAction acquireSegmentAction = nextLoadableSegment.acquire(acquireMode);
Review Comment:
It's been clear in testing that good overall performance on cold data is highly dependent on loadahead. With this change, we aren't fetching the entire segment as part of loadahead, which is going to lead to poorer download rates. It's ok to deal with this in a follow-up, but it does need to be dealt with somehow.
Either the query logic should declare what parts of the segment it wants to start with, or we should do something statistical: for each query, observe which parts of the segment are actually used, and fetch those during loadahead for future segments. The statistical approach only works for queries that fetch a large number of segments (so the earlier ones can be used to inform the later ones). But this is fine, since it's those queries where the performance concern lies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis merged PR #19535:
URL: https://github.com/apache/druid/pull/19535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3348717808
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle BEFORE submitting downloads. This sparse-allocates the bundle's container files
+ // and reserves the disk-usage accounting at bundle granularity. The returned Closeable releases the hold when
+ // the cursor closes, allowing the cache to later evict the bundle if needed.
+ // PartialBundleAcquirer.acquire() throws DruidException on failure.
+ final Closeable bundleHold = bundleAcquirer.acquire(bundleName);
+ // Release the bundle hold at most once: the canceler, the success-path holder close, and the failure callback can
+ // all race to release it.
+ final AtomicBoolean holdReleased = new AtomicBoolean(false);
+ final Closeable releaseHoldOnce = () -> {
+ if (holdReleased.compareAndSet(false, true)) {
+ bundleHold.close();
+ }
+ };
+
+ try {
+ // Submit one materialization task per column so a multi-threaded download executor can fan them out
+ final ListeningExecutorService downloadExec = bundleAcquirer.getDownloadExec();
+ final List<ListenableFuture<?>> columnDownloads = new ArrayList<>(requiredColumns.size());
+ for (String column : requiredColumns) {
+ columnDownloads.add(downloadExec.submit(() -> {
+ rowSelector.getColumnHolder(column);
+ return null;
+ }));
+ }
+
+ // Canceler runs if the awaiter closes this holder before it's ready (e.g. query cancel/timeout). cancel(true)
+ // stops column downloads that haven't begun their deep-storage read yet: queued tasks are skipped, and tasks
+ // parked on the download executor's permit are interrupted out of the (interruptible) wait before doing any I/O.
+ // A download already in its read/write loop runs to completion. After canceling, release the bundle hold. Once
+ // the holder is produced and handed to set(), ownership transfers to the awaiter, which drains it via
+ // close() (cancel) or release() (success); the once-guard keeps the hold release safe across all of these paths.
+ final AsyncCursorHolder asyncHolder = new AsyncCursorHolder(() -> {
+ for (ListenableFuture<?> columnDownload : columnDownloads) {
+ columnDownload.cancel(true);
+ }
+ CloseableUtils.closeAndSuppressExceptions(releaseHoldOnce, ignored -> {});
Review Comment:
[P1] Keep bundle holds until running downloads finish
When an async cursor is closed after a column download has already entered rowSelector.getColumnHolder/mapFile, cancel(true) does not wait for that callable to stop; the code even documents that in-flight read/write loops may run to completion. The canceler still releases the bundle hold immediately, which drops the bundle reference and storage hold while mapper work may still be using the container. That allows SIEVE/ephemeral eviction or a pending unmount to reach PartialSegmentFileMapperV10.evictContainer despite mapFile/ensureFilesAvailable still being in flight, violating that mapper's concurrency contract and risking deletion/unmap of a container being read or written. Defer releasing the bundle hold/reference until submitted downloads have actually returned, or add per-download protection so eviction cannot run while a started download continues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415470249
##########
processing/src/main/java/org/apache/druid/segment/PartialBundleAcquirer.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.ListeningExecutorService;
+
+import java.io.Closeable;
+
+/**
+ * Context object that bundles together pieces of cache-layer machinery needed to do on-demand partial segment loading:
+ * <ol>
+ * <li>A hook for acquiring a hold on the cache-layer bundle.</li>
+ * <li>The executor on which partial downloads should run.</li>
+ * </ol>
+ */
+public interface PartialBundleAcquirer
+{
+ /**
+ * Acquire a hold on the cache-layer bundle with the given name. The returned {@link Closeable} releases the hold
Review Comment:
see https://github.com/apache/druid/pull/19535#discussion_r3415467548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415427779
##########
server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:
##########
@@ -536,16 +897,30 @@ private AcquireSegmentAction acquireExistingSegment(SegmentCacheEntryIdentifier
location.addWeakReservationHoldIfExists(identifier)
);
if (hold != null) {
- if (hold.getEntry().isMounted()) {
+ if (!(hold.getEntry() instanceof CompleteSegmentCacheEntry complete)) {
+ // The eager (complete) acquire path found a non-complete entry under this id. This only arises if
+ // partial-load on-disk state survived a toggle of druid.segmentCache.virtualStoragePartialDownloadsEnabled
+ // to false (getCachedSegments reserves an on-disk partial layout regardless of the flag). The eager path
+ // cannot serve a partial layout; surface a clear operator error rather than a ClassCastException.
+ throw DruidException.forPersona(DruidException.Persona.OPERATOR)
Review Comment:
agree this was a wack failure mode, I have reworked stuff so that if we detect partials on disk during startup and partials are not enabled we now delete them instead of exploding at query time. I have also switched the config to disable this partial load functionality until we have some more time to mature things.
With regards to rollback to older versions, what happens is that on startup mount will fail with an exception because the path will not contain expected v9 or v10 segment files, and the handling on that exception will trigger deletion because the path exists _unless_ `druid.segmentCache.deleteOnRemove` is false (it is true by default). We could do something kind of tricky to ensure that they are always deleted, which is _always_ write the `DOWNLOAD_START_MARKER_FILE_NAME` for partial segments so it would trigger the 'corrupted segment' handling if we tried to mount as a regular segment, but I have not done that yet in this PR because it feels kind of strange.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "capistrant (via GitHub)" <gi...@apache.org>.
capistrant commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3404916556
##########
processing/src/main/java/org/apache/druid/segment/V10TimeBoundaryInspector.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.projections.ProjectionMetadata;
+import org.joda.time.DateTime;
+import org.joda.time.Interval;
+
+import javax.annotation.Nullable;
+
+/**
+ * {@link TimeBoundaryInspector} for V10 segments. Reads min/max {@link ColumnHolder#TIME_COLUMN_NAME} values from the
+ * base projection's {@link ProjectionMetadata} (persisted at write time by {@link IndexMergerBase}) so the inspector
+ * can answer without reading the {@code __time} column. Falls back to the segment's data interval (inexact) when the
+ * writer didn't supply min/max, for older V10 segments produced before the field existed.
+ * <p>
+ * The exact-bounds path is independent of segment sort order: the writer records the actual minimum and maximum
+ * timestamps across all walked rows even when the segment is not time-sorted. This is a strict improvement over
+ * the eager-load {@link QueryableIndexTimeBoundaryInspector}, which reports inexact bounds for non-time-sorted
+ * segments because its only option is to read positions 0 and N-1 of the {@code __time} column.
+ */
+public final class V10TimeBoundaryInspector implements TimeBoundaryInspector
+{
+ /**
+ * Build an inspector that reads from the given base projection metadata, falling back to {@code fallbackInterval}
+ * when the writer didn't persist min/max times (e.g. pre-Phase-1.5 segments).
Review Comment:
"pre-Phase-1.5" callout feels unnecessary with no added context
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.asyncresource.AsyncResource;
+import org.apache.druid.common.asyncresource.AsyncResources;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle before submitting downloads. Released on the cancel/failure paths (requestRelease)
+ // or handed to the produced cursor holder on success (releaser), exactly once, and never while an in-flight column
+ // download is mid-mapFile(). See BundleHoldRelease.
+ final BundleHoldRelease holdRelease = new BundleHoldRelease(bundleAcquirer.acquire(bundleName));
+
+ try {
+ // submit one materialization task per column so a multi-threaded download executor can fan them out
+ final List<AsyncResource<String>> columnDownloads = new ArrayList<>(requiredColumns.size());
+ for (String column : requiredColumns) {
+ columnDownloads.add(submitColumnDownload(rowSelector, column, holdRelease));
+ }
+ final AsyncResource<List<String>> downloaded = AsyncResources.collect(columnDownloads);
+
+ // Canceler runs if the awaiter closes this holder before it's ready (e.g. query cancel/timeout). Close the
+ // collected resource to cancel every column download that hasn't begun its deep-storage read yet (queued tasks
+ // are skipped; tasks parked on the download executor's permit are interrupted out of the interruptible wait
+ // before doing any I/O), then request the hold release through the handshake. Once the holder is produced and
+ // handed to set(), ownership transfers to the awaiter, which drains it via close() (cancel) or release()
+ // (success); the once-guard keeps the hold release safe across all of these paths.
+ final AsyncCursorHolder asyncHolder = new AsyncCursorHolder(() -> {
+ CloseableUtils.closeAndSuppressExceptions(downloaded, ignored -> {});
+ holdRelease.requestRelease();
+ });
+ downloaded.addReadyCallback(() -> {
+ final CursorHolder holder;
+ try {
+ downloaded.get(); // surfaces any column-download failure (or cancellation) as the cause
+ final CursorHolder inner = delegate.makeCursorHolderForProjection(spec, matched);
+ // The produced holder takes ownership of both inner and the bundle hold; from here, closing it releases both
+ holder = wrapWithBundleRelease(inner, holdRelease.releaser());
+ }
+ catch (Throwable t) {
+ // A column download failed or was canceled, this branch can fire while a sibling download is still
+ // mid-mapFile(), so the hold release must go through the handshake rather than dropping it here directly.
+ holdRelease.requestRelease();
+ asyncHolder.setException(t);
+ return;
+ }
+ if (!asyncHolder.set(holder)) {
+ // wrapper was closed (awaiter canceled) while we were producing the holder; close it ourselves so the
+ // holder, its inner, and the bundle hold don't leak.
+ holder.close();
+ }
+ });
+ return asyncHolder;
+ }
+ catch (Throwable t) {
+ // Failure between acquire and wiring up the downloads (submitDownload shut-down rejection, etc.). Ownership of
+ // the bundle hold hasn't transferred to the holder yet, so release it here.
+ throw CloseableUtils.closeAndWrapInCatch(t, holdRelease.releaser());
Review Comment:
why do we use `releaser()` here, which is documented as a "success-path" releaser, instead of requesting a release? also, do we need to worry about what is in `columnDownloads` at all at this point?
##########
server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:
##########
@@ -536,16 +897,30 @@ private AcquireSegmentAction acquireExistingSegment(SegmentCacheEntryIdentifier
location.addWeakReservationHoldIfExists(identifier)
);
if (hold != null) {
- if (hold.getEntry().isMounted()) {
+ if (!(hold.getEntry() instanceof CompleteSegmentCacheEntry complete)) {
+ // The eager (complete) acquire path found a non-complete entry under this id. This only arises if
+ // partial-load on-disk state survived a toggle of druid.segmentCache.virtualStoragePartialDownloadsEnabled
+ // to false (getCachedSegments reserves an on-disk partial layout regardless of the flag). The eager path
+ // cannot serve a partial layout; surface a clear operator error rather than a ClassCastException.
+ throw DruidException.forPersona(DruidException.Persona.OPERATOR)
Review Comment:
I am also interested in these ^
##########
processing/src/test/java/org/apache/druid/segment/PartialQueryableIndexCursorFactoryTest.java:
##########
@@ -0,0 +1,806 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import com.google.common.util.concurrent.SettableFuture;
+import org.apache.druid.common.asyncresource.AsyncResource;
+import org.apache.druid.common.asyncresource.AsyncResources;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.ListBasedInputRow;
+import org.apache.druid.data.input.impl.AggregateProjectionSpec;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.LongDimensionSchema;
+import org.apache.druid.data.input.impl.StringDimensionSchema;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import org.apache.druid.query.filter.EqualityFilter;
+import org.apache.druid.segment.column.ColumnConfig;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.CompressionStrategy;
+import org.apache.druid.segment.file.CountingRangeReader;
+import org.apache.druid.segment.file.PartialSegmentFileMapperV10;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.ReentrantLock;
+
+class PartialQueryableIndexCursorFactoryTest extends InitializedNullHandlingTest
+{
+ private static final ColumnConfig COLUMN_CONFIG = ColumnConfig.DEFAULT;
+ private static final DateTime TIME = DateTimes.of("2025-01-01");
+ private static final String PROJECTION_NAME = "dim1_metric1_sum";
+
+ private static final RowSignature ROW_SIGNATURE = RowSignature.builder()
+ .add("dim1", ColumnType.STRING)
+ .add("metric1", ColumnType.LONG)
+ .build();
+
+ private static final List<AggregateProjectionSpec> PROJECTIONS = Collections.singletonList(
+ AggregateProjectionSpec.builder(PROJECTION_NAME)
+ .groupingColumns(new StringDimensionSchema("dim1"))
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+
+ private static final List<InputRow> ROWS = Arrays.asList(
+ new ListBasedInputRow(ROW_SIGNATURE, TIME, ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 1L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(1), ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 2L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(2), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 3L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(3), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 4L))
+ );
+
+ @TempDir
+ static File sharedTempDir;
+
+ private static File segmentDir;
+ private static ListeningExecutorService realExec;
+
+ @TempDir
+ File perTestTempDir;
+
+ @BeforeAll
+ static void buildSegment()
+ {
+ final File tmpDir = new File(sharedTempDir, "build_" + ThreadLocalRandom.current().nextInt());
+ segmentDir = IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmpDir)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(PROJECTIONS)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder().withMetadataCompression(CompressionStrategy.NONE).build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+
+ realExec = MoreExecutors.listeningDecorator(Execs.singleThreaded("partial-cursor-test-%d"));
+ }
+
+ @AfterAll
+ static void teardownExec()
+ {
+ realExec.shutdownNow();
+ }
+
+ private record IndexAndMapper(PartialQueryableIndex index, PartialSegmentFileMapperV10 mapper)
+ implements AutoCloseable
+ {
+ @Override
+ public void close()
+ {
+ mapper.close();
+ }
+ }
+
+ private IndexAndMapper openIndex(CountingRangeReader rangeReader, String cacheName) throws IOException
+ {
+ final File cacheDir = new File(perTestTempDir, cacheName);
+ FileUtils.mkdirp(cacheDir);
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ TestHelper.makeJsonMapper(),
+ cacheDir,
+ IndexIO.V10_FILE_NAME,
+ Collections.emptyList()
+ );
+ return new IndexAndMapper(
+ new PartialQueryableIndex(mapper.getSegmentFileMetadata(), mapper, COLUMN_CONFIG),
+ mapper
+ );
+ }
+
+ private static ListeningExecutorService directExec()
+ {
+ return MoreExecutors.listeningDecorator(MoreExecutors.newDirectExecutorService());
+ }
+
+ private static PartialBundleAcquirer noOpAcquirer(ListeningExecutorService downloadExec)
+ {
+ return new PartialBundleAcquirer()
+ {
+ @Override
+ public Closeable acquire(String bundleName)
+ {
+ return () -> {};
+ }
+
+ @Override
+ public <T> AsyncResource<T> submitDownload(Callable<T> task)
+ {
+ return AsyncResources.fromFutureUnmanaged(downloadExec.submit(task));
+ }
+ };
+ }
+
+ @Test
+ void testSyncMakeCursorHolderThrowsWhenSegmentNotFullyDownloaded() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_throws")) {
+ final PartialQueryableIndex index = opened.index();
+ Assertions.assertFalse(index.isFullyDownloaded());
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Fresh mapper has only the header on disk; no internal files have been downloaded. The sync path must
+ // refuse rather than trigger downloads on the calling thread.
+ Assertions.assertThrows(
+ DruidException.class,
+ () -> factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)
+ );
+ }
+ }
+
+ @Test
+ void testSyncMakeCursorHolderSucceedsAfterFullDownload() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_after_async")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Eagerly materialize the entire segment via the file mapper to simulate the eager-acquire path.
+ mapper.ensureFilesAvailable(mapper.getSegmentFileMetadata().getFiles().keySet());
+ Assertions.assertTrue(mapper.isFullyDownloaded(), "test precondition: every internal file should be on disk");
+
+ try (CursorHolder holder = factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)) {
+ Assertions.assertNotNull(holder);
+ }
+ }
+ }
+
+ @Test
+ void testAsyncMakeCursorHolderDefersDownloadUntilExecutorRuns() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ final CountDownLatch gate = new CountDownLatch(1);
+ final ExecutorService rawExec = Execs.singleThreaded("partial-cursor-defer-%d");
+ final ListeningExecutorService gatedExec = MoreExecutors.listeningDecorator(rawExec);
+ try (IndexAndMapper opened = openIndex(rangeReader, "async_defer")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(gatedExec)
+ );
+ rangeReader.resetCount();
+
+ // Hold the executor with a pre-queued blocker so the actual download task can't start.
+ // assign the blocker future to an (intentionally unused) local so errorprone's CheckReturnValue is satisfied
+ @SuppressWarnings("unused")
+ ListenableFuture<?> unused = gatedExec.submit(() -> {
+ try {
+ gate.await();
+ }
+ catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ });
+
+ final AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(CursorBuildSpec.FULL_SCAN);
+ Assertions.assertFalse(
+ asyncHolder.isReady(),
+ "async holder must not be ready while download executor is still blocked"
+ );
+ Assertions.assertEquals(0, rangeReader.getReadCount(), "no download should have started yet");
+
+ final CountDownLatch ready = new CountDownLatch(1);
+ asyncHolder.addReadyCallback(ready::countDown);
+
+ // Release the gate; download should now run.
+ gate.countDown();
+ Assertions.assertTrue(ready.await(5, TimeUnit.SECONDS), "ready callback must fire after executor processes task");
+ Assertions.assertTrue(asyncHolder.isReady());
+ Assertions.assertTrue(rangeReader.getReadCount() > 0, "download must have happened by the time holder is ready");
+
+ try (CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ }
+ }
+ finally {
+ gatedExec.shutdownNow();
+ rawExec.shutdownNow();
+ }
+ }
+
+ @Test
+ void testMatchedProjectionDownloadsOnlyRequestedColumns() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "proj_columns")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Query asks for group-by dim1 + sum(metric1), but NOT count. The projection has dim1 / _metric1_sum / _count;
+ // the projection match rewrites this to physical columns {dim1, _metric1_sum}. We should see those download,
+ // but NOT the _count file; proves column-level pruning, not just projection-level.
+ final CursorBuildSpec aggSpec = CursorBuildSpec.builder()
+ .setGroupingColumns(List.of("dim1"))
+ .setAggregators(List.of(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1")
+ ))
+ .setPhysicalColumns(Set.of("dim1", "metric1"))
+ .build();
+
+ // Async path triggers the column-level pre-download. directExec makes the future complete synchronously.
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(aggSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ final String projPrefix = PROJECTION_NAME + "/";
+ final String basePrefix = Projections.BASE_TABLE_PROJECTION_NAME + "/";
+ final Set<String> downloaded = mapper.getDownloadedFiles();
+
+ // Requested projection columns materialized.
+ Assertions.assertTrue(downloaded.contains(projPrefix + "dim1"), "expected projection dim1; got: " + downloaded);
+ Assertions.assertTrue(
+ downloaded.contains(projPrefix + "_metric1_sum"),
+ "expected projection _metric1_sum; got: " + downloaded
+ );
+ // The projection's _count file was NOT in the query; must not be downloaded
+ Assertions.assertFalse(
+ downloaded.contains(projPrefix + "_count"),
+ "expected projection _count NOT to be downloaded; got: " + downloaded
+ );
+ // Base __time and base metric1 are NOT touched: projection dim1 may pull base dim1 as its parent column
+ // (legitimate dependency), but unrelated base columns must stay untouched.
+ Assertions.assertFalse(
+ downloaded.contains(basePrefix + ColumnHolder.TIME_COLUMN_NAME),
+ "expected base __time NOT to be downloaded; got: " + downloaded
+ );
+ Assertions.assertFalse(
+ downloaded.contains(basePrefix + "metric1"),
+ "expected base metric1 NOT to be downloaded; got: " + downloaded
+ );
+ }
+ }
+ }
+
+ @Test
+ void testOpeningTimeOrderedProjectionCursorTriggersNoDownload() throws IOException
+ {
+ // A projection grouped by [__gran(HOUR), dim1] is time-ordered, so QueryableIndexCursorHolder reads the
+ // projection's (downloadable) time column when it builds the cursor (interval-checking offset + the projection's
+ // own time-boundary inspector). A GROUP BY dim1 query matches the projection by re-aggregating over __gran and
+ // does NOT list __time in physicalColumns. requiredColumns must nonetheless pre-fetch __time so that read happens
+ // during the async pre-fetch rather than as a lazy download when the cursor is opened on a processing thread.
+ final String projectionName = "hourly_dim1_metric1_sum";
+ final File timeOrderedSegmentDir = buildTimeOrderedProjectionSegment(projectionName);
+
+ final CountingRangeReader rangeReader = new CountingRangeReader(timeOrderedSegmentDir);
+ final File cacheDir = new File(perTestTempDir, "time_ordered_proj");
+ FileUtils.mkdirp(cacheDir);
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ TestHelper.makeJsonMapper(),
+ cacheDir,
+ IndexIO.V10_FILE_NAME,
+ Collections.emptyList()
+ );
+ try {
+ final PartialQueryableIndex index =
+ new PartialQueryableIndex(mapper.getSegmentFileMetadata(), mapper, COLUMN_CONFIG);
+
+ // Precondition: the matched projection's row selector really is time-ordered, so cursor build WOULD read its
+ // time column. Without this, the assertion below would be vacuously satisfied.
+ final QueryableIndex projectionIndex = index.getProjectionQueryableIndex(projectionName);
+ Assertions.assertNotNull(projectionIndex, "test setup: projection should be present");
+ Assertions.assertEquals(
+ Order.ASCENDING,
+ Cursors.getTimeOrdering(projectionIndex.getOrdering()),
+ "test setup: projection must be time-ordered for this test to guard anything"
+ );
+
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+
+ // ETERNITY interval so the offset is not clipped to a residual interval (a genuine reason to read __time);
+ // here the only __time read is the time-ordered offset/inspector during cursor build, which the pre-fetch
+ // must have already covered. physicalColumns deliberately omits __time.
+ final CursorBuildSpec aggSpec = CursorBuildSpec.builder()
+ .setInterval(Intervals.ETERNITY)
+ .setGroupingColumns(List.of("dim1"))
+ .setAggregators(List.of(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1")
+ ))
+ .setPhysicalColumns(Set.of("dim1", "metric1"))
+ .build();
+
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(aggSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+
+ // After the async pre-fetch resolves, the projection's required columns AND __time are on disk, even though
+ // __time was not declared in physicalColumns.
+ final String projPrefix = projectionName + "/";
+ final Set<String> beforeOpen = mapper.getDownloadedFiles();
+ Assertions.assertTrue(
+ beforeOpen.contains(projPrefix + "_metric1_sum"),
+ "test setup: projection path should have run and pre-fetched _metric1_sum; got: " + beforeOpen
+ );
+ // The projection's time column is stored under its granularity grouping name (__gran); it is exposed as
+ // __time but the on-disk smoosh file (and thus the downloaded-files entry) is __gran.
+ Assertions.assertTrue(
+ beforeOpen.contains(projPrefix + "__gran"),
+ "requiredColumns must pre-fetch the projection's time column for a time-ordered cursor; got: " + beforeOpen
+ );
+
+ // Opening the cursor reads __time (interval-checking offset + the projection's time-boundary inspector). It
+ // was pre-fetched above, so this must not trigger any further (lazy) download on the processing thread.
+ Assertions.assertNotNull(holder.asCursor());
+
+ final Set<String> afterOpen = mapper.getDownloadedFiles();
+ Assertions.assertEquals(
+ beforeOpen,
+ afterOpen,
+ "opening a projection-matched cursor must not trigger a lazy download; newly downloaded: "
+ + Sets.difference(afterOpen, beforeOpen)
+ );
+ }
+ }
+ finally {
+ mapper.close();
+ }
+ }
+
+ @Test
+ void testOpeningTimeOrderedBaseCursorTriggersNoDownload() throws IOException
+ {
+ // The base table is time-ordered, so QueryableIndexCursorHolder reads __time when building the cursor (for its
+ // interval-checking offset) even though a raw scan does not list __time in physicalColumns. requiredColumns must
+ // pre-fetch __time so that read does not become a lazy download on the processing thread at cursor-open time.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "base_open_no_download")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ Assertions.assertEquals(
+ Order.ASCENDING,
+ Cursors.getTimeOrdering(index.getOrdering()),
+ "test setup: base table must be time-ordered"
+ );
+
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+
+ // Raw scan (no grouping/aggregators) so the aggregate projection is not matched and we exercise the base path.
+ // __time is deliberately omitted from physicalColumns.
+ final CursorBuildSpec scanSpec = CursorBuildSpec.builder()
+ .setInterval(Intervals.ETERNITY)
+ .setPhysicalColumns(Set.of("dim1"))
+ .build();
+
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(scanSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+
+ final String basePrefix = Projections.BASE_TABLE_PROJECTION_NAME + "/";
+ final Set<String> beforeOpen = mapper.getDownloadedFiles();
+ // base path ran (dim1 pre-fetched) and __time was pre-fetched despite not being in physicalColumns
+ Assertions.assertTrue(
+ beforeOpen.contains(basePrefix + "dim1"),
+ "test setup: base path should have pre-fetched dim1; got: " + beforeOpen
+ );
+ Assertions.assertTrue(
+ beforeOpen.contains(basePrefix + ColumnHolder.TIME_COLUMN_NAME),
+ "requiredColumns must pre-fetch __time for a time-ordered base cursor; got: " + beforeOpen
+ );
+
+ Assertions.assertNotNull(holder.asCursor());
+
+ final Set<String> afterOpen = mapper.getDownloadedFiles();
+ Assertions.assertEquals(
+ beforeOpen,
+ afterOpen,
+ "opening a time-ordered base cursor must not trigger a lazy download; newly downloaded: "
+ + Sets.difference(afterOpen, beforeOpen)
+ );
+ }
+ }
+ }
+
+ @Test
+ void testFilteredWrapperOverPartialUsesAsyncPath() throws IOException
+ {
+ // A FilteredCursorFactory wrapping a not-fully-downloaded partial cursor factory must route makeCursorHolderAsync
+ // through the delegate's async (download-on-demand) path. If FilteredCursorFactory fell back to the CursorFactory
+ // default makeCursorHolderAsync (which calls sync makeCursorHolder), the partial delegate would throw because the
+ // segment isn't fully downloaded.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "filtered_wrapper_async")) {
+ final PartialQueryableIndex index = opened.index();
+ Assertions.assertFalse(index.isFullyDownloaded(), "test setup: segment must not be fully downloaded");
+
+ final PartialQueryableIndexCursorFactory partial = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+ final CursorFactory filtered = new FilteredCursorFactory(
+ partial,
+ new EqualityFilter("dim1", ColumnType.STRING, "a", null)
+ );
+
+ // directExec resolves the async download inline; release() returns the built holder. asCursor() exercises the
+ // cursor over the on-demand-loaded columns. Would throw here if the wrapper used the sync default.
+ try (AsyncCursorHolder asyncHolder = filtered.makeCursorHolderAsync(CursorBuildSpec.FULL_SCAN);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ Assertions.assertNotNull(holder.asCursor(), "async path should build a usable cursor without a sync download");
+ }
+ }
+ }
+
+ private File buildTimeOrderedProjectionSegment(String projectionName)
Review Comment:
nit: private helper method interleaved within tests
##########
processing/src/test/java/org/apache/druid/segment/PartialQueryableIndexCursorFactoryTest.java:
##########
@@ -0,0 +1,806 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import com.google.common.util.concurrent.SettableFuture;
+import org.apache.druid.common.asyncresource.AsyncResource;
+import org.apache.druid.common.asyncresource.AsyncResources;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.ListBasedInputRow;
+import org.apache.druid.data.input.impl.AggregateProjectionSpec;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.LongDimensionSchema;
+import org.apache.druid.data.input.impl.StringDimensionSchema;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import org.apache.druid.query.filter.EqualityFilter;
+import org.apache.druid.segment.column.ColumnConfig;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.CompressionStrategy;
+import org.apache.druid.segment.file.CountingRangeReader;
+import org.apache.druid.segment.file.PartialSegmentFileMapperV10;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.ReentrantLock;
+
+class PartialQueryableIndexCursorFactoryTest extends InitializedNullHandlingTest
+{
+ private static final ColumnConfig COLUMN_CONFIG = ColumnConfig.DEFAULT;
+ private static final DateTime TIME = DateTimes.of("2025-01-01");
+ private static final String PROJECTION_NAME = "dim1_metric1_sum";
+
+ private static final RowSignature ROW_SIGNATURE = RowSignature.builder()
+ .add("dim1", ColumnType.STRING)
+ .add("metric1", ColumnType.LONG)
+ .build();
+
+ private static final List<AggregateProjectionSpec> PROJECTIONS = Collections.singletonList(
+ AggregateProjectionSpec.builder(PROJECTION_NAME)
+ .groupingColumns(new StringDimensionSchema("dim1"))
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+
+ private static final List<InputRow> ROWS = Arrays.asList(
+ new ListBasedInputRow(ROW_SIGNATURE, TIME, ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 1L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(1), ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 2L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(2), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 3L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(3), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 4L))
+ );
+
+ @TempDir
+ static File sharedTempDir;
+
+ private static File segmentDir;
+ private static ListeningExecutorService realExec;
+
+ @TempDir
+ File perTestTempDir;
+
+ @BeforeAll
+ static void buildSegment()
+ {
+ final File tmpDir = new File(sharedTempDir, "build_" + ThreadLocalRandom.current().nextInt());
+ segmentDir = IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmpDir)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(PROJECTIONS)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder().withMetadataCompression(CompressionStrategy.NONE).build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+
+ realExec = MoreExecutors.listeningDecorator(Execs.singleThreaded("partial-cursor-test-%d"));
+ }
+
+ @AfterAll
+ static void teardownExec()
+ {
+ realExec.shutdownNow();
+ }
+
+ private record IndexAndMapper(PartialQueryableIndex index, PartialSegmentFileMapperV10 mapper)
+ implements AutoCloseable
+ {
+ @Override
+ public void close()
+ {
+ mapper.close();
+ }
+ }
+
+ private IndexAndMapper openIndex(CountingRangeReader rangeReader, String cacheName) throws IOException
+ {
+ final File cacheDir = new File(perTestTempDir, cacheName);
+ FileUtils.mkdirp(cacheDir);
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ TestHelper.makeJsonMapper(),
+ cacheDir,
+ IndexIO.V10_FILE_NAME,
+ Collections.emptyList()
+ );
+ return new IndexAndMapper(
+ new PartialQueryableIndex(mapper.getSegmentFileMetadata(), mapper, COLUMN_CONFIG),
+ mapper
+ );
+ }
+
+ private static ListeningExecutorService directExec()
+ {
+ return MoreExecutors.listeningDecorator(MoreExecutors.newDirectExecutorService());
+ }
+
+ private static PartialBundleAcquirer noOpAcquirer(ListeningExecutorService downloadExec)
+ {
+ return new PartialBundleAcquirer()
+ {
+ @Override
+ public Closeable acquire(String bundleName)
+ {
+ return () -> {};
+ }
+
+ @Override
+ public <T> AsyncResource<T> submitDownload(Callable<T> task)
+ {
+ return AsyncResources.fromFutureUnmanaged(downloadExec.submit(task));
+ }
+ };
+ }
+
+ @Test
+ void testSyncMakeCursorHolderThrowsWhenSegmentNotFullyDownloaded() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_throws")) {
+ final PartialQueryableIndex index = opened.index();
+ Assertions.assertFalse(index.isFullyDownloaded());
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Fresh mapper has only the header on disk; no internal files have been downloaded. The sync path must
+ // refuse rather than trigger downloads on the calling thread.
+ Assertions.assertThrows(
+ DruidException.class,
+ () -> factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)
+ );
+ }
+ }
+
+ @Test
+ void testSyncMakeCursorHolderSucceedsAfterFullDownload() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_after_async")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Eagerly materialize the entire segment via the file mapper to simulate the eager-acquire path.
+ mapper.ensureFilesAvailable(mapper.getSegmentFileMetadata().getFiles().keySet());
+ Assertions.assertTrue(mapper.isFullyDownloaded(), "test precondition: every internal file should be on disk");
+
+ try (CursorHolder holder = factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)) {
+ Assertions.assertNotNull(holder);
+ }
+ }
+ }
+
+ @Test
+ void testAsyncMakeCursorHolderDefersDownloadUntilExecutorRuns() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ final CountDownLatch gate = new CountDownLatch(1);
+ final ExecutorService rawExec = Execs.singleThreaded("partial-cursor-defer-%d");
+ final ListeningExecutorService gatedExec = MoreExecutors.listeningDecorator(rawExec);
+ try (IndexAndMapper opened = openIndex(rangeReader, "async_defer")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(gatedExec)
+ );
+ rangeReader.resetCount();
+
+ // Hold the executor with a pre-queued blocker so the actual download task can't start.
+ // assign the blocker future to an (intentionally unused) local so errorprone's CheckReturnValue is satisfied
+ @SuppressWarnings("unused")
+ ListenableFuture<?> unused = gatedExec.submit(() -> {
+ try {
+ gate.await();
+ }
+ catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ });
+
+ final AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(CursorBuildSpec.FULL_SCAN);
+ Assertions.assertFalse(
+ asyncHolder.isReady(),
+ "async holder must not be ready while download executor is still blocked"
+ );
+ Assertions.assertEquals(0, rangeReader.getReadCount(), "no download should have started yet");
+
+ final CountDownLatch ready = new CountDownLatch(1);
+ asyncHolder.addReadyCallback(ready::countDown);
+
+ // Release the gate; download should now run.
+ gate.countDown();
+ Assertions.assertTrue(ready.await(5, TimeUnit.SECONDS), "ready callback must fire after executor processes task");
+ Assertions.assertTrue(asyncHolder.isReady());
+ Assertions.assertTrue(rangeReader.getReadCount() > 0, "download must have happened by the time holder is ready");
+
+ try (CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ }
+ }
+ finally {
+ gatedExec.shutdownNow();
+ rawExec.shutdownNow();
+ }
+ }
+
+ @Test
+ void testMatchedProjectionDownloadsOnlyRequestedColumns() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "proj_columns")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Query asks for group-by dim1 + sum(metric1), but NOT count. The projection has dim1 / _metric1_sum / _count;
+ // the projection match rewrites this to physical columns {dim1, _metric1_sum}. We should see those download,
+ // but NOT the _count file; proves column-level pruning, not just projection-level.
+ final CursorBuildSpec aggSpec = CursorBuildSpec.builder()
+ .setGroupingColumns(List.of("dim1"))
+ .setAggregators(List.of(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1")
+ ))
+ .setPhysicalColumns(Set.of("dim1", "metric1"))
+ .build();
+
+ // Async path triggers the column-level pre-download. directExec makes the future complete synchronously.
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(aggSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ final String projPrefix = PROJECTION_NAME + "/";
+ final String basePrefix = Projections.BASE_TABLE_PROJECTION_NAME + "/";
+ final Set<String> downloaded = mapper.getDownloadedFiles();
+
+ // Requested projection columns materialized.
+ Assertions.assertTrue(downloaded.contains(projPrefix + "dim1"), "expected projection dim1; got: " + downloaded);
+ Assertions.assertTrue(
+ downloaded.contains(projPrefix + "_metric1_sum"),
+ "expected projection _metric1_sum; got: " + downloaded
+ );
+ // The projection's _count file was NOT in the query; must not be downloaded
+ Assertions.assertFalse(
+ downloaded.contains(projPrefix + "_count"),
+ "expected projection _count NOT to be downloaded; got: " + downloaded
+ );
+ // Base __time and base metric1 are NOT touched: projection dim1 may pull base dim1 as its parent column
+ // (legitimate dependency), but unrelated base columns must stay untouched.
+ Assertions.assertFalse(
+ downloaded.contains(basePrefix + ColumnHolder.TIME_COLUMN_NAME),
+ "expected base __time NOT to be downloaded; got: " + downloaded
+ );
+ Assertions.assertFalse(
+ downloaded.contains(basePrefix + "metric1"),
+ "expected base metric1 NOT to be downloaded; got: " + downloaded
+ );
+ }
+ }
+ }
+
+ @Test
+ void testOpeningTimeOrderedProjectionCursorTriggersNoDownload() throws IOException
+ {
+ // A projection grouped by [__gran(HOUR), dim1] is time-ordered, so QueryableIndexCursorHolder reads the
+ // projection's (downloadable) time column when it builds the cursor (interval-checking offset + the projection's
+ // own time-boundary inspector). A GROUP BY dim1 query matches the projection by re-aggregating over __gran and
+ // does NOT list __time in physicalColumns. requiredColumns must nonetheless pre-fetch __time so that read happens
+ // during the async pre-fetch rather than as a lazy download when the cursor is opened on a processing thread.
+ final String projectionName = "hourly_dim1_metric1_sum";
+ final File timeOrderedSegmentDir = buildTimeOrderedProjectionSegment(projectionName);
+
+ final CountingRangeReader rangeReader = new CountingRangeReader(timeOrderedSegmentDir);
+ final File cacheDir = new File(perTestTempDir, "time_ordered_proj");
+ FileUtils.mkdirp(cacheDir);
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ TestHelper.makeJsonMapper(),
+ cacheDir,
+ IndexIO.V10_FILE_NAME,
+ Collections.emptyList()
+ );
+ try {
+ final PartialQueryableIndex index =
+ new PartialQueryableIndex(mapper.getSegmentFileMetadata(), mapper, COLUMN_CONFIG);
+
+ // Precondition: the matched projection's row selector really is time-ordered, so cursor build WOULD read its
+ // time column. Without this, the assertion below would be vacuously satisfied.
+ final QueryableIndex projectionIndex = index.getProjectionQueryableIndex(projectionName);
+ Assertions.assertNotNull(projectionIndex, "test setup: projection should be present");
+ Assertions.assertEquals(
+ Order.ASCENDING,
+ Cursors.getTimeOrdering(projectionIndex.getOrdering()),
+ "test setup: projection must be time-ordered for this test to guard anything"
+ );
+
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+
+ // ETERNITY interval so the offset is not clipped to a residual interval (a genuine reason to read __time);
+ // here the only __time read is the time-ordered offset/inspector during cursor build, which the pre-fetch
+ // must have already covered. physicalColumns deliberately omits __time.
+ final CursorBuildSpec aggSpec = CursorBuildSpec.builder()
+ .setInterval(Intervals.ETERNITY)
+ .setGroupingColumns(List.of("dim1"))
+ .setAggregators(List.of(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1")
+ ))
+ .setPhysicalColumns(Set.of("dim1", "metric1"))
+ .build();
+
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(aggSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+
+ // After the async pre-fetch resolves, the projection's required columns AND __time are on disk, even though
+ // __time was not declared in physicalColumns.
+ final String projPrefix = projectionName + "/";
+ final Set<String> beforeOpen = mapper.getDownloadedFiles();
+ Assertions.assertTrue(
+ beforeOpen.contains(projPrefix + "_metric1_sum"),
+ "test setup: projection path should have run and pre-fetched _metric1_sum; got: " + beforeOpen
+ );
+ // The projection's time column is stored under its granularity grouping name (__gran); it is exposed as
+ // __time but the on-disk smoosh file (and thus the downloaded-files entry) is __gran.
+ Assertions.assertTrue(
+ beforeOpen.contains(projPrefix + "__gran"),
+ "requiredColumns must pre-fetch the projection's time column for a time-ordered cursor; got: " + beforeOpen
+ );
+
+ // Opening the cursor reads __time (interval-checking offset + the projection's time-boundary inspector). It
+ // was pre-fetched above, so this must not trigger any further (lazy) download on the processing thread.
+ Assertions.assertNotNull(holder.asCursor());
+
+ final Set<String> afterOpen = mapper.getDownloadedFiles();
+ Assertions.assertEquals(
+ beforeOpen,
+ afterOpen,
+ "opening a projection-matched cursor must not trigger a lazy download; newly downloaded: "
+ + Sets.difference(afterOpen, beforeOpen)
+ );
+ }
+ }
+ finally {
+ mapper.close();
+ }
+ }
+
+ @Test
+ void testOpeningTimeOrderedBaseCursorTriggersNoDownload() throws IOException
+ {
+ // The base table is time-ordered, so QueryableIndexCursorHolder reads __time when building the cursor (for its
+ // interval-checking offset) even though a raw scan does not list __time in physicalColumns. requiredColumns must
+ // pre-fetch __time so that read does not become a lazy download on the processing thread at cursor-open time.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "base_open_no_download")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ Assertions.assertEquals(
+ Order.ASCENDING,
+ Cursors.getTimeOrdering(index.getOrdering()),
+ "test setup: base table must be time-ordered"
+ );
+
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+
+ // Raw scan (no grouping/aggregators) so the aggregate projection is not matched and we exercise the base path.
+ // __time is deliberately omitted from physicalColumns.
+ final CursorBuildSpec scanSpec = CursorBuildSpec.builder()
+ .setInterval(Intervals.ETERNITY)
+ .setPhysicalColumns(Set.of("dim1"))
+ .build();
+
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(scanSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+
+ final String basePrefix = Projections.BASE_TABLE_PROJECTION_NAME + "/";
+ final Set<String> beforeOpen = mapper.getDownloadedFiles();
+ // base path ran (dim1 pre-fetched) and __time was pre-fetched despite not being in physicalColumns
+ Assertions.assertTrue(
+ beforeOpen.contains(basePrefix + "dim1"),
+ "test setup: base path should have pre-fetched dim1; got: " + beforeOpen
+ );
+ Assertions.assertTrue(
+ beforeOpen.contains(basePrefix + ColumnHolder.TIME_COLUMN_NAME),
+ "requiredColumns must pre-fetch __time for a time-ordered base cursor; got: " + beforeOpen
+ );
+
+ Assertions.assertNotNull(holder.asCursor());
+
+ final Set<String> afterOpen = mapper.getDownloadedFiles();
+ Assertions.assertEquals(
+ beforeOpen,
+ afterOpen,
+ "opening a time-ordered base cursor must not trigger a lazy download; newly downloaded: "
+ + Sets.difference(afterOpen, beforeOpen)
+ );
+ }
+ }
+ }
+
+ @Test
+ void testFilteredWrapperOverPartialUsesAsyncPath() throws IOException
+ {
+ // A FilteredCursorFactory wrapping a not-fully-downloaded partial cursor factory must route makeCursorHolderAsync
+ // through the delegate's async (download-on-demand) path. If FilteredCursorFactory fell back to the CursorFactory
+ // default makeCursorHolderAsync (which calls sync makeCursorHolder), the partial delegate would throw because the
+ // segment isn't fully downloaded.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "filtered_wrapper_async")) {
+ final PartialQueryableIndex index = opened.index();
+ Assertions.assertFalse(index.isFullyDownloaded(), "test setup: segment must not be fully downloaded");
+
+ final PartialQueryableIndexCursorFactory partial = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+ final CursorFactory filtered = new FilteredCursorFactory(
+ partial,
+ new EqualityFilter("dim1", ColumnType.STRING, "a", null)
+ );
+
+ // directExec resolves the async download inline; release() returns the built holder. asCursor() exercises the
+ // cursor over the on-demand-loaded columns. Would throw here if the wrapper used the sync default.
+ try (AsyncCursorHolder asyncHolder = filtered.makeCursorHolderAsync(CursorBuildSpec.FULL_SCAN);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ Assertions.assertNotNull(holder.asCursor(), "async path should build a usable cursor without a sync download");
+ }
+ }
+ }
+
+ private File buildTimeOrderedProjectionSegment(String projectionName)
+ {
+ final List<AggregateProjectionSpec> projections = Collections.singletonList(
+ AggregateProjectionSpec.builder(projectionName)
+ .virtualColumns(Granularities.toVirtualColumn(Granularities.HOUR, "__gran"))
+ .groupingColumns(
+ new LongDimensionSchema("__gran"),
+ new StringDimensionSchema("dim1")
+ )
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+ final File tmpDir = new File(perTestTempDir, "build_time_ordered_" + ThreadLocalRandom.current().nextInt(Integer.MAX_VALUE));
+ return IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmpDir)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(projections)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder().withMetadataCompression(CompressionStrategy.NONE).build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+ }
+
+ @Test
+ void testRowSignatureAndCapabilitiesDelegatedToUnderlyingFactory() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "delegate")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // delegating to QueryableIndexCursorFactory which returns the index's row signature
+ final RowSignature signature = factory.getRowSignature();
+ Assertions.assertNotNull(signature);
+ Assertions.assertTrue(signature.size() > 0);
+
+ Assertions.assertNotNull(factory.getColumnCapabilities("dim1"));
+ }
+ }
+
+ @Test
+ void testAsyncCloseBeforeDownloadCompletesClosesCursorHolderOnce() throws Exception
+ {
+ // Producer-doesn't-leak invariant: if the wrapper is closed before the producer's set(holder) fires, the producer
+ // is responsible for closing the holder. Use the real (single-threaded) executor with a gate that pauses the
+ // download task before it builds the cursor, then close the wrapper, then release the gate.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ final AtomicReference<RuntimeException> bgErr = new AtomicReference<>();
Review Comment:
bgErr doesn't actually seem to be wired into the test at all. it is just created and used in get assertion below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415507135
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle BEFORE submitting downloads. This sparse-allocates the bundle's container files
+ // and reserves the disk-usage accounting at bundle granularity. The returned Closeable releases the hold when
+ // the cursor closes, allowing the cache to later evict the bundle if needed.
+ // PartialBundleAcquirer.acquire() throws DruidException on failure.
+ final Closeable bundleHold = bundleAcquirer.acquire(bundleName);
+
+ try {
+ // Submit one materialization task per column so a multi-threaded download executor can fan them out
+ final ListeningExecutorService downloadExec = bundleAcquirer.getDownloadExec();
+ final List<ListenableFuture<?>> columnDownloads = new ArrayList<>(requiredColumns.size());
+ for (String column : requiredColumns) {
+ columnDownloads.add(downloadExec.submit(() -> {
+ rowSelector.getColumnHolder(column);
+ return null;
+ }));
+ }
+
+ // No canceler: downloads are allowed to complete even if the awaiter cancels. Once they complete and the inner
+ // CursorHolder is handed to asyncHolder.set(), it's owned by the asyncHolder until the awaiter takes one of
+ // two paths: asyncHolder.close() (cancel, closes the held holder) or asyncHolder.release() (success, transfers
+ // ownership to the caller). Either way the holder is drained and nothing leaks.
+ final AsyncCursorHolder asyncHolder = new AsyncCursorHolder(null);
+ Futures.addCallback(
+ Futures.allAsList(columnDownloads),
Review Comment:
switched this to use `collect`, not `transform` though yet since still trying to figure out how to make the transformed thing a closeable, safely. I think we would need a new `transformCloseable` or .. something to handle this case since the current transform only closes the source, not the output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415524774
##########
processing/src/test/java/org/apache/druid/segment/PartialQueryableIndexCursorFactoryTest.java:
##########
@@ -0,0 +1,806 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import com.google.common.util.concurrent.SettableFuture;
+import org.apache.druid.common.asyncresource.AsyncResource;
+import org.apache.druid.common.asyncresource.AsyncResources;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.ListBasedInputRow;
+import org.apache.druid.data.input.impl.AggregateProjectionSpec;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.LongDimensionSchema;
+import org.apache.druid.data.input.impl.StringDimensionSchema;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import org.apache.druid.query.filter.EqualityFilter;
+import org.apache.druid.segment.column.ColumnConfig;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.CompressionStrategy;
+import org.apache.druid.segment.file.CountingRangeReader;
+import org.apache.druid.segment.file.PartialSegmentFileMapperV10;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.ReentrantLock;
+
+class PartialQueryableIndexCursorFactoryTest extends InitializedNullHandlingTest
+{
+ private static final ColumnConfig COLUMN_CONFIG = ColumnConfig.DEFAULT;
+ private static final DateTime TIME = DateTimes.of("2025-01-01");
+ private static final String PROJECTION_NAME = "dim1_metric1_sum";
+
+ private static final RowSignature ROW_SIGNATURE = RowSignature.builder()
+ .add("dim1", ColumnType.STRING)
+ .add("metric1", ColumnType.LONG)
+ .build();
+
+ private static final List<AggregateProjectionSpec> PROJECTIONS = Collections.singletonList(
+ AggregateProjectionSpec.builder(PROJECTION_NAME)
+ .groupingColumns(new StringDimensionSchema("dim1"))
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+
+ private static final List<InputRow> ROWS = Arrays.asList(
+ new ListBasedInputRow(ROW_SIGNATURE, TIME, ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 1L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(1), ROW_SIGNATURE.getColumnNames(), Arrays.asList("a", 2L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(2), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 3L)),
+ new ListBasedInputRow(ROW_SIGNATURE, TIME.plusMinutes(3), ROW_SIGNATURE.getColumnNames(), Arrays.asList("b", 4L))
+ );
+
+ @TempDir
+ static File sharedTempDir;
+
+ private static File segmentDir;
+ private static ListeningExecutorService realExec;
+
+ @TempDir
+ File perTestTempDir;
+
+ @BeforeAll
+ static void buildSegment()
+ {
+ final File tmpDir = new File(sharedTempDir, "build_" + ThreadLocalRandom.current().nextInt());
+ segmentDir = IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmpDir)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(PROJECTIONS)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder().withMetadataCompression(CompressionStrategy.NONE).build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+
+ realExec = MoreExecutors.listeningDecorator(Execs.singleThreaded("partial-cursor-test-%d"));
+ }
+
+ @AfterAll
+ static void teardownExec()
+ {
+ realExec.shutdownNow();
+ }
+
+ private record IndexAndMapper(PartialQueryableIndex index, PartialSegmentFileMapperV10 mapper)
+ implements AutoCloseable
+ {
+ @Override
+ public void close()
+ {
+ mapper.close();
+ }
+ }
+
+ private IndexAndMapper openIndex(CountingRangeReader rangeReader, String cacheName) throws IOException
+ {
+ final File cacheDir = new File(perTestTempDir, cacheName);
+ FileUtils.mkdirp(cacheDir);
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ TestHelper.makeJsonMapper(),
+ cacheDir,
+ IndexIO.V10_FILE_NAME,
+ Collections.emptyList()
+ );
+ return new IndexAndMapper(
+ new PartialQueryableIndex(mapper.getSegmentFileMetadata(), mapper, COLUMN_CONFIG),
+ mapper
+ );
+ }
+
+ private static ListeningExecutorService directExec()
+ {
+ return MoreExecutors.listeningDecorator(MoreExecutors.newDirectExecutorService());
+ }
+
+ private static PartialBundleAcquirer noOpAcquirer(ListeningExecutorService downloadExec)
+ {
+ return new PartialBundleAcquirer()
+ {
+ @Override
+ public Closeable acquire(String bundleName)
+ {
+ return () -> {};
+ }
+
+ @Override
+ public <T> AsyncResource<T> submitDownload(Callable<T> task)
+ {
+ return AsyncResources.fromFutureUnmanaged(downloadExec.submit(task));
+ }
+ };
+ }
+
+ @Test
+ void testSyncMakeCursorHolderThrowsWhenSegmentNotFullyDownloaded() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_throws")) {
+ final PartialQueryableIndex index = opened.index();
+ Assertions.assertFalse(index.isFullyDownloaded());
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Fresh mapper has only the header on disk; no internal files have been downloaded. The sync path must
+ // refuse rather than trigger downloads on the calling thread.
+ Assertions.assertThrows(
+ DruidException.class,
+ () -> factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)
+ );
+ }
+ }
+
+ @Test
+ void testSyncMakeCursorHolderSucceedsAfterFullDownload() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "sync_after_async")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Eagerly materialize the entire segment via the file mapper to simulate the eager-acquire path.
+ mapper.ensureFilesAvailable(mapper.getSegmentFileMetadata().getFiles().keySet());
+ Assertions.assertTrue(mapper.isFullyDownloaded(), "test precondition: every internal file should be on disk");
+
+ try (CursorHolder holder = factory.makeCursorHolder(CursorBuildSpec.FULL_SCAN)) {
+ Assertions.assertNotNull(holder);
+ }
+ }
+ }
+
+ @Test
+ void testAsyncMakeCursorHolderDefersDownloadUntilExecutorRuns() throws Exception
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ final CountDownLatch gate = new CountDownLatch(1);
+ final ExecutorService rawExec = Execs.singleThreaded("partial-cursor-defer-%d");
+ final ListeningExecutorService gatedExec = MoreExecutors.listeningDecorator(rawExec);
+ try (IndexAndMapper opened = openIndex(rangeReader, "async_defer")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(gatedExec)
+ );
+ rangeReader.resetCount();
+
+ // Hold the executor with a pre-queued blocker so the actual download task can't start.
+ // assign the blocker future to an (intentionally unused) local so errorprone's CheckReturnValue is satisfied
+ @SuppressWarnings("unused")
+ ListenableFuture<?> unused = gatedExec.submit(() -> {
+ try {
+ gate.await();
+ }
+ catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ });
+
+ final AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(CursorBuildSpec.FULL_SCAN);
+ Assertions.assertFalse(
+ asyncHolder.isReady(),
+ "async holder must not be ready while download executor is still blocked"
+ );
+ Assertions.assertEquals(0, rangeReader.getReadCount(), "no download should have started yet");
+
+ final CountDownLatch ready = new CountDownLatch(1);
+ asyncHolder.addReadyCallback(ready::countDown);
+
+ // Release the gate; download should now run.
+ gate.countDown();
+ Assertions.assertTrue(ready.await(5, TimeUnit.SECONDS), "ready callback must fire after executor processes task");
+ Assertions.assertTrue(asyncHolder.isReady());
+ Assertions.assertTrue(rangeReader.getReadCount() > 0, "download must have happened by the time holder is ready");
+
+ try (CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ }
+ }
+ finally {
+ gatedExec.shutdownNow();
+ rawExec.shutdownNow();
+ }
+ }
+
+ @Test
+ void testMatchedProjectionDownloadsOnlyRequestedColumns() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "proj_columns")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // Query asks for group-by dim1 + sum(metric1), but NOT count. The projection has dim1 / _metric1_sum / _count;
+ // the projection match rewrites this to physical columns {dim1, _metric1_sum}. We should see those download,
+ // but NOT the _count file; proves column-level pruning, not just projection-level.
+ final CursorBuildSpec aggSpec = CursorBuildSpec.builder()
+ .setGroupingColumns(List.of("dim1"))
+ .setAggregators(List.of(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1")
+ ))
+ .setPhysicalColumns(Set.of("dim1", "metric1"))
+ .build();
+
+ // Async path triggers the column-level pre-download. directExec makes the future complete synchronously.
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(aggSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ final String projPrefix = PROJECTION_NAME + "/";
+ final String basePrefix = Projections.BASE_TABLE_PROJECTION_NAME + "/";
+ final Set<String> downloaded = mapper.getDownloadedFiles();
+
+ // Requested projection columns materialized.
+ Assertions.assertTrue(downloaded.contains(projPrefix + "dim1"), "expected projection dim1; got: " + downloaded);
+ Assertions.assertTrue(
+ downloaded.contains(projPrefix + "_metric1_sum"),
+ "expected projection _metric1_sum; got: " + downloaded
+ );
+ // The projection's _count file was NOT in the query; must not be downloaded
+ Assertions.assertFalse(
+ downloaded.contains(projPrefix + "_count"),
+ "expected projection _count NOT to be downloaded; got: " + downloaded
+ );
+ // Base __time and base metric1 are NOT touched: projection dim1 may pull base dim1 as its parent column
+ // (legitimate dependency), but unrelated base columns must stay untouched.
+ Assertions.assertFalse(
+ downloaded.contains(basePrefix + ColumnHolder.TIME_COLUMN_NAME),
+ "expected base __time NOT to be downloaded; got: " + downloaded
+ );
+ Assertions.assertFalse(
+ downloaded.contains(basePrefix + "metric1"),
+ "expected base metric1 NOT to be downloaded; got: " + downloaded
+ );
+ }
+ }
+ }
+
+ @Test
+ void testOpeningTimeOrderedProjectionCursorTriggersNoDownload() throws IOException
+ {
+ // A projection grouped by [__gran(HOUR), dim1] is time-ordered, so QueryableIndexCursorHolder reads the
+ // projection's (downloadable) time column when it builds the cursor (interval-checking offset + the projection's
+ // own time-boundary inspector). A GROUP BY dim1 query matches the projection by re-aggregating over __gran and
+ // does NOT list __time in physicalColumns. requiredColumns must nonetheless pre-fetch __time so that read happens
+ // during the async pre-fetch rather than as a lazy download when the cursor is opened on a processing thread.
+ final String projectionName = "hourly_dim1_metric1_sum";
+ final File timeOrderedSegmentDir = buildTimeOrderedProjectionSegment(projectionName);
+
+ final CountingRangeReader rangeReader = new CountingRangeReader(timeOrderedSegmentDir);
+ final File cacheDir = new File(perTestTempDir, "time_ordered_proj");
+ FileUtils.mkdirp(cacheDir);
+ final PartialSegmentFileMapperV10 mapper = PartialSegmentFileMapperV10.create(
+ rangeReader,
+ TestHelper.makeJsonMapper(),
+ cacheDir,
+ IndexIO.V10_FILE_NAME,
+ Collections.emptyList()
+ );
+ try {
+ final PartialQueryableIndex index =
+ new PartialQueryableIndex(mapper.getSegmentFileMetadata(), mapper, COLUMN_CONFIG);
+
+ // Precondition: the matched projection's row selector really is time-ordered, so cursor build WOULD read its
+ // time column. Without this, the assertion below would be vacuously satisfied.
+ final QueryableIndex projectionIndex = index.getProjectionQueryableIndex(projectionName);
+ Assertions.assertNotNull(projectionIndex, "test setup: projection should be present");
+ Assertions.assertEquals(
+ Order.ASCENDING,
+ Cursors.getTimeOrdering(projectionIndex.getOrdering()),
+ "test setup: projection must be time-ordered for this test to guard anything"
+ );
+
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+
+ // ETERNITY interval so the offset is not clipped to a residual interval (a genuine reason to read __time);
+ // here the only __time read is the time-ordered offset/inspector during cursor build, which the pre-fetch
+ // must have already covered. physicalColumns deliberately omits __time.
+ final CursorBuildSpec aggSpec = CursorBuildSpec.builder()
+ .setInterval(Intervals.ETERNITY)
+ .setGroupingColumns(List.of("dim1"))
+ .setAggregators(List.of(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1")
+ ))
+ .setPhysicalColumns(Set.of("dim1", "metric1"))
+ .build();
+
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(aggSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+
+ // After the async pre-fetch resolves, the projection's required columns AND __time are on disk, even though
+ // __time was not declared in physicalColumns.
+ final String projPrefix = projectionName + "/";
+ final Set<String> beforeOpen = mapper.getDownloadedFiles();
+ Assertions.assertTrue(
+ beforeOpen.contains(projPrefix + "_metric1_sum"),
+ "test setup: projection path should have run and pre-fetched _metric1_sum; got: " + beforeOpen
+ );
+ // The projection's time column is stored under its granularity grouping name (__gran); it is exposed as
+ // __time but the on-disk smoosh file (and thus the downloaded-files entry) is __gran.
+ Assertions.assertTrue(
+ beforeOpen.contains(projPrefix + "__gran"),
+ "requiredColumns must pre-fetch the projection's time column for a time-ordered cursor; got: " + beforeOpen
+ );
+
+ // Opening the cursor reads __time (interval-checking offset + the projection's time-boundary inspector). It
+ // was pre-fetched above, so this must not trigger any further (lazy) download on the processing thread.
+ Assertions.assertNotNull(holder.asCursor());
+
+ final Set<String> afterOpen = mapper.getDownloadedFiles();
+ Assertions.assertEquals(
+ beforeOpen,
+ afterOpen,
+ "opening a projection-matched cursor must not trigger a lazy download; newly downloaded: "
+ + Sets.difference(afterOpen, beforeOpen)
+ );
+ }
+ }
+ finally {
+ mapper.close();
+ }
+ }
+
+ @Test
+ void testOpeningTimeOrderedBaseCursorTriggersNoDownload() throws IOException
+ {
+ // The base table is time-ordered, so QueryableIndexCursorHolder reads __time when building the cursor (for its
+ // interval-checking offset) even though a raw scan does not list __time in physicalColumns. requiredColumns must
+ // pre-fetch __time so that read does not become a lazy download on the processing thread at cursor-open time.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "base_open_no_download")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialSegmentFileMapperV10 mapper = opened.mapper();
+ Assertions.assertEquals(
+ Order.ASCENDING,
+ Cursors.getTimeOrdering(index.getOrdering()),
+ "test setup: base table must be time-ordered"
+ );
+
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+
+ // Raw scan (no grouping/aggregators) so the aggregate projection is not matched and we exercise the base path.
+ // __time is deliberately omitted from physicalColumns.
+ final CursorBuildSpec scanSpec = CursorBuildSpec.builder()
+ .setInterval(Intervals.ETERNITY)
+ .setPhysicalColumns(Set.of("dim1"))
+ .build();
+
+ try (AsyncCursorHolder asyncHolder = factory.makeCursorHolderAsync(scanSpec);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+
+ final String basePrefix = Projections.BASE_TABLE_PROJECTION_NAME + "/";
+ final Set<String> beforeOpen = mapper.getDownloadedFiles();
+ // base path ran (dim1 pre-fetched) and __time was pre-fetched despite not being in physicalColumns
+ Assertions.assertTrue(
+ beforeOpen.contains(basePrefix + "dim1"),
+ "test setup: base path should have pre-fetched dim1; got: " + beforeOpen
+ );
+ Assertions.assertTrue(
+ beforeOpen.contains(basePrefix + ColumnHolder.TIME_COLUMN_NAME),
+ "requiredColumns must pre-fetch __time for a time-ordered base cursor; got: " + beforeOpen
+ );
+
+ Assertions.assertNotNull(holder.asCursor());
+
+ final Set<String> afterOpen = mapper.getDownloadedFiles();
+ Assertions.assertEquals(
+ beforeOpen,
+ afterOpen,
+ "opening a time-ordered base cursor must not trigger a lazy download; newly downloaded: "
+ + Sets.difference(afterOpen, beforeOpen)
+ );
+ }
+ }
+ }
+
+ @Test
+ void testFilteredWrapperOverPartialUsesAsyncPath() throws IOException
+ {
+ // A FilteredCursorFactory wrapping a not-fully-downloaded partial cursor factory must route makeCursorHolderAsync
+ // through the delegate's async (download-on-demand) path. If FilteredCursorFactory fell back to the CursorFactory
+ // default makeCursorHolderAsync (which calls sync makeCursorHolder), the partial delegate would throw because the
+ // segment isn't fully downloaded.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "filtered_wrapper_async")) {
+ final PartialQueryableIndex index = opened.index();
+ Assertions.assertFalse(index.isFullyDownloaded(), "test setup: segment must not be fully downloaded");
+
+ final PartialQueryableIndexCursorFactory partial = new PartialQueryableIndexCursorFactory(
+ index,
+ V10TimeBoundaryInspector.forBaseProjection(index.getBaseProjectionMetadata(), index.getDataInterval()),
+ noOpAcquirer(directExec())
+ );
+ final CursorFactory filtered = new FilteredCursorFactory(
+ partial,
+ new EqualityFilter("dim1", ColumnType.STRING, "a", null)
+ );
+
+ // directExec resolves the async download inline; release() returns the built holder. asCursor() exercises the
+ // cursor over the on-demand-loaded columns. Would throw here if the wrapper used the sync default.
+ try (AsyncCursorHolder asyncHolder = filtered.makeCursorHolderAsync(CursorBuildSpec.FULL_SCAN);
+ CursorHolder holder = asyncHolder.release()) {
+ Assertions.assertNotNull(holder);
+ Assertions.assertNotNull(holder.asCursor(), "async path should build a usable cursor without a sync download");
+ }
+ }
+ }
+
+ private File buildTimeOrderedProjectionSegment(String projectionName)
+ {
+ final List<AggregateProjectionSpec> projections = Collections.singletonList(
+ AggregateProjectionSpec.builder(projectionName)
+ .virtualColumns(Granularities.toVirtualColumn(Granularities.HOUR, "__gran"))
+ .groupingColumns(
+ new LongDimensionSchema("__gran"),
+ new StringDimensionSchema("dim1")
+ )
+ .aggregators(
+ new LongSumAggregatorFactory("_metric1_sum", "metric1"),
+ new CountAggregatorFactory("_count")
+ )
+ .build()
+ );
+ final File tmpDir = new File(perTestTempDir, "build_time_ordered_" + ThreadLocalRandom.current().nextInt(Integer.MAX_VALUE));
+ return IndexBuilder.create()
+ .useV10()
+ .tmpDir(tmpDir)
+ .segmentWriteOutMediumFactory(OffHeapMemorySegmentWriteOutMediumFactory.instance())
+ .schema(
+ IncrementalIndexSchema.builder()
+ .withDimensionsSpec(
+ DimensionsSpec.builder()
+ .setDimensions(
+ List.of(
+ new StringDimensionSchema("dim1"),
+ new LongDimensionSchema("metric1")
+ )
+ )
+ .build()
+ )
+ .withRollup(false)
+ .withMinTimestamp(TIME.getMillis())
+ .withProjections(projections)
+ .build()
+ )
+ .indexSpec(IndexSpec.builder().withMetadataCompression(CompressionStrategy.NONE).build())
+ .rows(ROWS)
+ .buildMMappedIndexFile();
+ }
+
+ @Test
+ void testRowSignatureAndCapabilitiesDelegatedToUnderlyingFactory() throws IOException
+ {
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ try (IndexAndMapper opened = openIndex(rangeReader, "delegate")) {
+ final PartialQueryableIndex index = opened.index();
+ final PartialQueryableIndexCursorFactory factory = new PartialQueryableIndexCursorFactory(
+ index,
+ QueryableIndexTimeBoundaryInspector.create(index),
+ noOpAcquirer(directExec())
+ );
+
+ // delegating to QueryableIndexCursorFactory which returns the index's row signature
+ final RowSignature signature = factory.getRowSignature();
+ Assertions.assertNotNull(signature);
+ Assertions.assertTrue(signature.size() > 0);
+
+ Assertions.assertNotNull(factory.getColumnCapabilities("dim1"));
+ }
+ }
+
+ @Test
+ void testAsyncCloseBeforeDownloadCompletesClosesCursorHolderOnce() throws Exception
+ {
+ // Producer-doesn't-leak invariant: if the wrapper is closed before the producer's set(holder) fires, the producer
+ // is responsible for closing the holder. Use the real (single-threaded) executor with a gate that pauses the
+ // download task before it builds the cursor, then close the wrapper, then release the gate.
+ final CountingRangeReader rangeReader = new CountingRangeReader(segmentDir);
+ final AtomicReference<RuntimeException> bgErr = new AtomicReference<>();
Review Comment:
oops, this was stale test after refactoring some stuff, it no longer had a point, deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415516060
##########
server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:
##########
@@ -526,6 +548,345 @@ public AcquireSegmentAction acquireSegment(final DataSegment dataSegment)
}
}
+ @Override
+ public Optional<Segment> acquireCachedPartialSegment(SegmentId segmentId)
+ {
+ if (!config.isVirtualStoragePartialDownloadsEnabled()) {
+ return acquireCachedSegment(segmentId);
+ }
+ return acquireCachedInternal(segmentId, false);
+ }
+
+ @Override
+ public AcquireSegmentAction acquirePartialSegment(DataSegment dataSegment)
+ {
+ final SegmentRangeReader rangeReader = tryOpenRangeReader(dataSegment);
+ if (rangeReader == null) {
+ // Storage backend doesn't support range reads for this segment (e.g. V9 format or zipped)
+ return acquireSegment(dataSegment);
+ }
+ return acquirePartialInternal(dataSegment, rangeReader, false);
+ }
+
+
+ /**
+ * Shared implementation for {@link #acquireCachedSegment} and {@link #acquireCachedPartialSegment}. Walks the
+ * storage locations checking for existing entries.
+ * <p>
+ * When {@code requireFullyDownloaded} is {@code true} the entry must also report
+ * {@link SegmentCacheEntry#isFullyDownloaded()} (the contract for {@link #acquireCachedSegment}), which never
+ * hands back an entry that isn't behaviorally fully-materialized. When {@code false} a mounted entry is
+ * sufficient ({@link #acquireCachedPartialSegment}'s lazy-mount contract), even if some bundles are still
+ * not yet on disk. In either case {@link SegmentCacheEntry#acquireReference(Closeable)} composes the weak-entry hold
+ * into the returned segment's close lifecycle.
+ */
+ private Optional<Segment> acquireCachedInternal(SegmentId segmentId, boolean requireFullyDownloaded)
+ {
+ final SegmentCacheEntryIdentifier id = new SegmentCacheEntryIdentifier(segmentId);
+ for (StorageLocation location : locations) {
+ final SegmentCacheEntry staticEntry = location.getStaticCacheEntry(id);
+ if (staticEntry != null) {
+ if (staticEntry.isMounted() && (!requireFullyDownloaded || staticEntry.isFullyDownloaded())) {
+ return staticEntry.acquireReference();
+ }
+ return Optional.empty();
+ }
+ if (!config.isVirtualStorage()) {
+ continue;
+ }
+ final StorageLocation.ReservationHold<SegmentCacheEntry> hold = location.addWeakReservationHoldIfExists(id);
+ if (hold != null) {
+ if (hold.getEntry().isMounted() && (!requireFullyDownloaded || hold.getEntry().isFullyDownloaded())) {
+ return hold.getEntry().acquireReference(hold);
+ }
+ hold.close();
+ return Optional.empty();
+ }
+ }
+ return Optional.empty();
+ }
+
+ /**
+ * Shared scaffolding for both partial acquire APIs. {@code fullDownload=false} powers
+ * {@link #acquirePartialSegment} (lazy mount, header bytes only; bundles mount on demand at query time);
+ * {@code fullDownload=true} powers the partial-eligible branch of {@link #acquireSegment} (mount +
+ * {@link PartialSegmentFileMapperV10#ensureAllDownloaded} so the returned segment is fully-materialized).
+ * <p>
+ * Fast path: {@link #findExistingPartialWithHold} locates an existing entry across locations under a hold. If the
+ * entry is already usable (mounted, and fully-downloaded when required), return an immediate-future action whose
+ * {@code loadCleanup} is the hold (the supplier mints fresh segments per call, each with its own metadata
+ * reference, so no separate cleanup is needed).
+ * <p>
+ * Slow path: under the per-segment lock, {@link #findOrReservePartial} reuses an existing not-yet-usable entry or
+ * reserves a fresh weak one, and the action submits mount (+ optional ensureAllDownloaded) to
+ * {@link #virtualStorageLoadOnDemandExec} so callers that yield on the future never block a processing thread on
+ * deep-storage I/O.
+ */
+ private AcquireSegmentAction acquirePartialInternal(
+ DataSegment dataSegment,
+ SegmentRangeReader rangeReader,
+ boolean fullDownload
Review Comment:
yea, this was an oversight in a refactor, it should be handled correctly now
##########
server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:
##########
@@ -526,6 +548,345 @@ public AcquireSegmentAction acquireSegment(final DataSegment dataSegment)
}
}
+ @Override
+ public Optional<Segment> acquireCachedPartialSegment(SegmentId segmentId)
+ {
+ if (!config.isVirtualStoragePartialDownloadsEnabled()) {
+ return acquireCachedSegment(segmentId);
+ }
+ return acquireCachedInternal(segmentId, false);
+ }
+
+ @Override
+ public AcquireSegmentAction acquirePartialSegment(DataSegment dataSegment)
+ {
+ final SegmentRangeReader rangeReader = tryOpenRangeReader(dataSegment);
+ if (rangeReader == null) {
+ // Storage backend doesn't support range reads for this segment (e.g. V9 format or zipped)
+ return acquireSegment(dataSegment);
+ }
+ return acquirePartialInternal(dataSegment, rangeReader, false);
+ }
+
+
+ /**
+ * Shared implementation for {@link #acquireCachedSegment} and {@link #acquireCachedPartialSegment}. Walks the
+ * storage locations checking for existing entries.
+ * <p>
+ * When {@code requireFullyDownloaded} is {@code true} the entry must also report
+ * {@link SegmentCacheEntry#isFullyDownloaded()} (the contract for {@link #acquireCachedSegment}), which never
+ * hands back an entry that isn't behaviorally fully-materialized. When {@code false} a mounted entry is
+ * sufficient ({@link #acquireCachedPartialSegment}'s lazy-mount contract), even if some bundles are still
+ * not yet on disk. In either case {@link SegmentCacheEntry#acquireReference(Closeable)} composes the weak-entry hold
+ * into the returned segment's close lifecycle.
+ */
+ private Optional<Segment> acquireCachedInternal(SegmentId segmentId, boolean requireFullyDownloaded)
+ {
+ final SegmentCacheEntryIdentifier id = new SegmentCacheEntryIdentifier(segmentId);
+ for (StorageLocation location : locations) {
+ final SegmentCacheEntry staticEntry = location.getStaticCacheEntry(id);
+ if (staticEntry != null) {
+ if (staticEntry.isMounted() && (!requireFullyDownloaded || staticEntry.isFullyDownloaded())) {
+ return staticEntry.acquireReference();
+ }
+ return Optional.empty();
+ }
+ if (!config.isVirtualStorage()) {
+ continue;
+ }
+ final StorageLocation.ReservationHold<SegmentCacheEntry> hold = location.addWeakReservationHoldIfExists(id);
+ if (hold != null) {
+ if (hold.getEntry().isMounted() && (!requireFullyDownloaded || hold.getEntry().isFullyDownloaded())) {
+ return hold.getEntry().acquireReference(hold);
+ }
+ hold.close();
+ return Optional.empty();
+ }
+ }
+ return Optional.empty();
+ }
+
+ /**
+ * Shared scaffolding for both partial acquire APIs. {@code fullDownload=false} powers
+ * {@link #acquirePartialSegment} (lazy mount, header bytes only; bundles mount on demand at query time);
+ * {@code fullDownload=true} powers the partial-eligible branch of {@link #acquireSegment} (mount +
+ * {@link PartialSegmentFileMapperV10#ensureAllDownloaded} so the returned segment is fully-materialized).
+ * <p>
+ * Fast path: {@link #findExistingPartialWithHold} locates an existing entry across locations under a hold. If the
+ * entry is already usable (mounted, and fully-downloaded when required), return an immediate-future action whose
+ * {@code loadCleanup} is the hold (the supplier mints fresh segments per call, each with its own metadata
+ * reference, so no separate cleanup is needed).
+ * <p>
+ * Slow path: under the per-segment lock, {@link #findOrReservePartial} reuses an existing not-yet-usable entry or
+ * reserves a fresh weak one, and the action submits mount (+ optional ensureAllDownloaded) to
+ * {@link #virtualStorageLoadOnDemandExec} so callers that yield on the future never block a processing thread on
+ * deep-storage I/O.
+ */
+ private AcquireSegmentAction acquirePartialInternal(
+ DataSegment dataSegment,
+ SegmentRangeReader rangeReader,
+ boolean fullDownload
Review Comment:
yea, this was an oversight in a refactor, good catch; it should be handled correctly now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415379015
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexSegment.java:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.utils.CloseableUtils;
+import org.joda.time.Interval;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.io.Closeable;
+
+/**
+ * {@link Segment} wrapper around a {@link PartialQueryableIndex}. Mirrors {@link QueryableIndexSegment} but wires up
+ * the V10-specific {@link V10TimeBoundaryInspector} (which answers from
+ * {@link org.apache.druid.segment.projections.ProjectionMetadata} min/max fields without downloading the
+ * {@code __time} column) and the partial-aware {@link PartialQueryableIndexCursorFactory} (which downloads
+ * required files on the supplied download executor before handing back a cursor).
+ * <p>
+ * Lifecycle: this segment is intended to exist as a transient reference-hold scope over an externally-owned
+ * {@link PartialQueryableIndex}, it never closes the underlying index. The {@code onClose} hook is what
+ * {@link #close()} fires when the segment is closed (e.g. to 'release' reference tracking stuff in the cache layer).
+ */
+public class PartialQueryableIndexSegment implements ReferenceCountedSegmentProvider.LeafReference
+{
+ private final PartialQueryableIndex index;
+ private final PartialQueryableIndexCursorFactory cursorFactory;
+ private final TimeBoundaryInspector timeBoundaryInspector;
+ private final SegmentId segmentId;
+ private final Closeable onClose;
+
+ public PartialQueryableIndexSegment(
+ PartialQueryableIndex index,
+ SegmentId segmentId,
+ Closeable onClose,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.timeBoundaryInspector = V10TimeBoundaryInspector.forBaseProjection(
+ index.getBaseProjectionMetadata(),
+ index.getDataInterval()
+ );
+ this.cursorFactory = new PartialQueryableIndexCursorFactory(
+ index,
+ timeBoundaryInspector,
+ bundleAcquirer
+ );
+ this.segmentId = segmentId;
+ this.onClose = onClose;
+ }
+
+ @Override
+ public SegmentId getId()
+ {
+ return segmentId;
+ }
+
+ @Override
+ public Interval getDataInterval()
+ {
+ return index.getDataInterval();
+ }
+
+ @Override
+ public void close()
+ {
+ CloseableUtils.closeAndWrapExceptions(onClose);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Nullable
+ @Override
+ public <T> T as(@Nonnull Class<T> clazz)
Review Comment:
#19577 split out `RowCountInspector` which is now wired up in this PR so we can answer the row count stuff which is much more widely used than the other parts of this interface (and answerable purely by metadata with v10 segments)
##########
server/src/main/java/org/apache/druid/server/SegmentManager.java:
##########
@@ -225,6 +224,32 @@ public AcquireSegmentAction acquireSegment(DataSegment dataSegment)
return cacheManager.acquireSegment(dataSegment);
}
+ /**
+ * Partial-load variant of {@link #acquireCachedSegment(SegmentId)}, returns a {@link Segment} when the cache holds
+ * an entry for the id; empty otherwise. The returned segment may not be fully loaded, callers must use async methods
+ * like {@link org.apache.druid.segment.CursorFactory#makeCursorHolderAsync} to download data on-demand. If the
+ * returned segment is only partially loaded, the synchronous methods like {@code makeCursorHolder} will fail if
+ * anything is still missing. If the segment is fully loaded, or not capable of partial loading, this method will
+ * still return a segment if it is present in cache and any async methods will function properly and return
+ * immediately.
+ */
+ public Optional<Segment> acquireCachedPartialSegment(SegmentId segmentId)
+ {
+ return cacheManager.acquireCachedPartialSegment(segmentId);
+ }
+
+ /**
+ * Partial-load variant of {@link #acquireSegment(DataSegment)}, returns an {@link AcquireSegmentAction} that
Review Comment:
the methods are now combined and accept an enum argument, i think it is a bit clearer what happens
##########
server/src/main/java/org/apache/druid/server/SegmentManager.java:
##########
@@ -225,6 +224,32 @@ public AcquireSegmentAction acquireSegment(DataSegment dataSegment)
return cacheManager.acquireSegment(dataSegment);
}
+ /**
+ * Partial-load variant of {@link #acquireCachedSegment(SegmentId)}, returns a {@link Segment} when the cache holds
+ * an entry for the id; empty otherwise. The returned segment may not be fully loaded, callers must use async methods
+ * like {@link org.apache.druid.segment.CursorFactory#makeCursorHolderAsync} to download data on-demand. If the
+ * returned segment is only partially loaded, the synchronous methods like {@code makeCursorHolder} will fail if
+ * anything is still missing. If the segment is fully loaded, or not capable of partial loading, this method will
+ * still return a segment if it is present in cache and any async methods will function properly and return
+ * immediately.
+ */
+ public Optional<Segment> acquireCachedPartialSegment(SegmentId segmentId)
+ {
+ return cacheManager.acquireCachedPartialSegment(segmentId);
+ }
+
+ /**
+ * Partial-load variant of {@link #acquireSegment(DataSegment)}, returns an {@link AcquireSegmentAction} that
Review Comment:
the methods are now combined and accept an `AcquireMode` enum argument, i think it is a bit clearer what happens
##########
server/src/main/java/org/apache/druid/server/SegmentManager.java:
##########
@@ -225,6 +224,32 @@ public AcquireSegmentAction acquireSegment(DataSegment dataSegment)
return cacheManager.acquireSegment(dataSegment);
}
+ /**
+ * Partial-load variant of {@link #acquireCachedSegment(SegmentId)}, returns a {@link Segment} when the cache holds
Review Comment:
same thing with the enum `AcquireMode` argument, i think it should be more obvious now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3415430556
##########
processing/src/main/java/org/apache/druid/segment/PartialBundleAcquirer.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.ListeningExecutorService;
+
+import java.io.Closeable;
+
+/**
+ * Context object that bundles together pieces of cache-layer machinery needed to do on-demand partial segment loading:
+ * <ol>
+ * <li>A hook for acquiring a hold on the cache-layer bundle.</li>
+ * <li>The executor on which partial downloads should run.</li>
+ * </ol>
+ */
+public interface PartialBundleAcquirer
+{
+ /**
+ * Acquire a hold on the cache-layer bundle with the given name. The returned {@link Closeable} releases the hold
+ * when closed (the bundle itself stays in the cache for re-use by subsequent acquires).
+ * <p>
+ * Implementations should be safe to call concurrently for different bundle names.
Review Comment:
clarified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3343345086
##########
server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:
##########
@@ -536,16 +897,30 @@ private AcquireSegmentAction acquireExistingSegment(SegmentCacheEntryIdentifier
location.addWeakReservationHoldIfExists(identifier)
);
if (hold != null) {
- if (hold.getEntry().isMounted()) {
+ if (!(hold.getEntry() instanceof CompleteSegmentCacheEntry complete)) {
+ // The eager (complete) acquire path found a non-complete entry under this id. This only arises if
+ // partial-load on-disk state survived a toggle of druid.segmentCache.virtualStoragePartialDownloadsEnabled
+ // to false (getCachedSegments reserves an on-disk partial layout regardless of the flag). The eager path
+ // cannot serve a partial layout; surface a clear operator error rather than a ClassCastException.
+ throw DruidException.forPersona(DruidException.Persona.OPERATOR)
Review Comment:
Two concerns:
1. Won't this also be an issue on downgrade?
2. Is it possible to do something that would cause the old version to repair this rather than throwing an error? As is, it will be difficult to turn off partial downloads once they are enabled.
##########
server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:
##########
@@ -536,16 +897,30 @@ private AcquireSegmentAction acquireExistingSegment(SegmentCacheEntryIdentifier
location.addWeakReservationHoldIfExists(identifier)
);
if (hold != null) {
- if (hold.getEntry().isMounted()) {
+ if (!(hold.getEntry() instanceof CompleteSegmentCacheEntry complete)) {
+ // The eager (complete) acquire path found a non-complete entry under this id. This only arises if
+ // partial-load on-disk state survived a toggle of druid.segmentCache.virtualStoragePartialDownloadsEnabled
+ // to false (getCachedSegments reserves an on-disk partial layout regardless of the flag). The eager path
+ // cannot serve a partial layout; surface a clear operator error rather than a ClassCastException.
+ throw DruidException.forPersona(DruidException.Persona.OPERATOR)
Review Comment:
Two concerns:
1. Won't this also be an issue on downgrade?
2. Is it possible to repair this rather than throwing an error? As is, it will be difficult to turn off partial downloads once they are enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: on-demand partial segment loading in SegmentLocalCacheManager (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #19535:
URL: https://github.com/apache/druid/pull/19535#discussion_r3338719602
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexSegment.java:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.utils.CloseableUtils;
+import org.joda.time.Interval;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.io.Closeable;
+
+/**
+ * {@link Segment} wrapper around a {@link PartialQueryableIndex}. Mirrors {@link QueryableIndexSegment} but wires up
+ * the V10-specific {@link V10TimeBoundaryInspector} (which answers from
+ * {@link org.apache.druid.segment.projections.ProjectionMetadata} min/max fields without downloading the
+ * {@code __time} column) and the partial-aware {@link PartialQueryableIndexCursorFactory} (which downloads
+ * required files on the supplied download executor before handing back a cursor).
+ * <p>
+ * Lifecycle: this segment is intended to exist as a transient reference-hold scope over an externally-owned
+ * {@link PartialQueryableIndex}, it never closes the underlying index. The {@code onClose} hook is what
+ * {@link #close()} fires when the segment is closed (e.g. to 'release' reference tracking stuff in the cache layer).
+ */
+public class PartialQueryableIndexSegment implements ReferenceCountedSegmentProvider.LeafReference
+{
+ private final PartialQueryableIndex index;
+ private final PartialQueryableIndexCursorFactory cursorFactory;
+ private final TimeBoundaryInspector timeBoundaryInspector;
+ private final SegmentId segmentId;
+ private final Closeable onClose;
+
+ public PartialQueryableIndexSegment(
+ PartialQueryableIndex index,
+ SegmentId segmentId,
+ Closeable onClose,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.timeBoundaryInspector = V10TimeBoundaryInspector.forBaseProjection(
+ index.getBaseProjectionMetadata(),
+ index.getDataInterval()
+ );
+ this.cursorFactory = new PartialQueryableIndexCursorFactory(
+ index,
+ timeBoundaryInspector,
+ bundleAcquirer
+ );
+ this.segmentId = segmentId;
+ this.onClose = onClose;
+ }
+
+ @Override
+ public SegmentId getId()
+ {
+ return segmentId;
+ }
+
+ @Override
+ public Interval getDataInterval()
+ {
+ return index.getDataInterval();
+ }
+
+ @Override
+ public void close()
+ {
+ CloseableUtils.closeAndWrapExceptions(onClose);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Nullable
+ @Override
+ public <T> T as(@Nonnull Class<T> clazz)
Review Comment:
Is it possible to include a `PhysicalSegmentInspector`? It would be useful to get the number of rows, which is used for incrementing certain counters.
##########
server/src/main/java/org/apache/druid/server/SegmentManager.java:
##########
@@ -225,6 +224,32 @@ public AcquireSegmentAction acquireSegment(DataSegment dataSegment)
return cacheManager.acquireSegment(dataSegment);
}
+ /**
+ * Partial-load variant of {@link #acquireCachedSegment(SegmentId)}, returns a {@link Segment} when the cache holds
Review Comment:
Is the segment returned by `acquireCachedSegment` guaranteed to be fully-loaded? (Its javadocs do not seem to make that clear.)
##########
server/src/main/java/org/apache/druid/server/SegmentManager.java:
##########
@@ -225,6 +224,32 @@ public AcquireSegmentAction acquireSegment(DataSegment dataSegment)
return cacheManager.acquireSegment(dataSegment);
}
+ /**
+ * Partial-load variant of {@link #acquireCachedSegment(SegmentId)}, returns a {@link Segment} when the cache holds
+ * an entry for the id; empty otherwise. The returned segment may not be fully loaded, callers must use async methods
+ * like {@link org.apache.druid.segment.CursorFactory#makeCursorHolderAsync} to download data on-demand. If the
+ * returned segment is only partially loaded, the synchronous methods like {@code makeCursorHolder} will fail if
+ * anything is still missing. If the segment is fully loaded, or not capable of partial loading, this method will
+ * still return a segment if it is present in cache and any async methods will function properly and return
+ * immediately.
+ */
+ public Optional<Segment> acquireCachedPartialSegment(SegmentId segmentId)
+ {
+ return cacheManager.acquireCachedPartialSegment(segmentId);
+ }
+
+ /**
+ * Partial-load variant of {@link #acquireSegment(DataSegment)}, returns an {@link AcquireSegmentAction} that
Review Comment:
Is the segment returned by `acquireSegment` guaranteed to be fully-loaded? (Its javadocs do not seem to make that clear.)
##########
server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:
##########
@@ -536,16 +897,30 @@ private AcquireSegmentAction acquireExistingSegment(SegmentCacheEntryIdentifier
location.addWeakReservationHoldIfExists(identifier)
);
if (hold != null) {
- if (hold.getEntry().isMounted()) {
+ if (!(hold.getEntry() instanceof CompleteSegmentCacheEntry complete)) {
+ // The eager (complete) acquire path found a non-complete entry under this id. This only arises if
+ // partial-load on-disk state survived a toggle of druid.segmentCache.virtualStoragePartialDownloadsEnabled
+ // to false (getCachedSegments reserves an on-disk partial layout regardless of the flag). The eager path
+ // cannot serve a partial layout; surface a clear operator error rather than a ClassCastException.
+ throw DruidException.forPersona(DruidException.Persona.OPERATOR)
Review Comment:
Two concerns:
1. Won't this also be an issue on downgrade?
2. Is it possible to repair this rather than throwing an error? As is, it will be difficult to turn off partial downloads once they are enabled.
##########
multi-stage-query/src/main/java/org/apache/druid/msq/input/RegularLoadableSegment.java:
##########
@@ -140,7 +140,7 @@ public synchronized Optional<Segment> acquireIfCached()
throw DruidException.defensive("Segment with descriptor[%s] is already acquired", descriptor);
}
- final Optional<Segment> cachedSegment = segmentManager.acquireCachedSegment(segmentId);
+ final Optional<Segment> cachedSegment = segmentManager.acquireCachedPartialSegment(segmentId);
Review Comment:
Make it an option whether `RegularLoadableSegment` does a partial fetch or full fetch? It may not be a safe assumption that all MSQ implementations of query processing are going to be able to handle partial segments / async stuff. Even if all existing ones do, there's also future ones, ones from extensions, etc.
##########
processing/src/main/java/org/apache/druid/segment/PartialBundleAcquirer.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.ListeningExecutorService;
+
+import java.io.Closeable;
+
+/**
+ * Context object that bundles together pieces of cache-layer machinery needed to do on-demand partial segment loading:
+ * <ol>
+ * <li>A hook for acquiring a hold on the cache-layer bundle.</li>
+ * <li>The executor on which partial downloads should run.</li>
+ * </ol>
+ */
+public interface PartialBundleAcquirer
+{
+ /**
+ * Acquire a hold on the cache-layer bundle with the given name. The returned {@link Closeable} releases the hold
Review Comment:
How are callers supposed to know what the bundle names are?
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle BEFORE submitting downloads. This sparse-allocates the bundle's container files
+ // and reserves the disk-usage accounting at bundle granularity. The returned Closeable releases the hold when
+ // the cursor closes, allowing the cache to later evict the bundle if needed.
+ // PartialBundleAcquirer.acquire() throws DruidException on failure.
Review Comment:
This comment is too long
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
Review Comment:
Given this is going to blow up, the default impl for `makeCursorHolderAsync` (which calls `makeCursorHolder`) isn't very useful, and seems likely to lead to delegating `CursorFactory` being implemented improperly. Consider deleting it, and perhaps replacing it with an abstract base class that can be used for leaf `CursorFactory` that don't have special logic for async cursor creation.
##########
processing/src/main/java/org/apache/druid/segment/PartialBundleAcquirer.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.ListeningExecutorService;
+
+import java.io.Closeable;
+
+/**
+ * Context object that bundles together pieces of cache-layer machinery needed to do on-demand partial segment loading:
+ * <ol>
+ * <li>A hook for acquiring a hold on the cache-layer bundle.</li>
+ * <li>The executor on which partial downloads should run.</li>
+ * </ol>
+ */
+public interface PartialBundleAcquirer
+{
+ /**
+ * Acquire a hold on the cache-layer bundle with the given name. The returned {@link Closeable} releases the hold
+ * when closed (the bundle itself stays in the cache for re-use by subsequent acquires).
+ * <p>
+ * Implementations should be safe to call concurrently for different bundle names.
Review Comment:
"should be" or "must be"? Does the wording imply it's *not* safe to call concurrently with the same bundle name?
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
Review Comment:
How did we know that the bundle name should be the projection name? Is that specified somewhere or just how things are currently implemented?
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexSegment.java:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.utils.CloseableUtils;
+import org.joda.time.Interval;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.io.Closeable;
+
+/**
+ * {@link Segment} wrapper around a {@link PartialQueryableIndex}. Mirrors {@link QueryableIndexSegment} but wires up
+ * the V10-specific {@link V10TimeBoundaryInspector} (which answers from
+ * {@link org.apache.druid.segment.projections.ProjectionMetadata} min/max fields without downloading the
+ * {@code __time} column) and the partial-aware {@link PartialQueryableIndexCursorFactory} (which downloads
+ * required files on the supplied download executor before handing back a cursor).
+ * <p>
+ * Lifecycle: this segment is intended to exist as a transient reference-hold scope over an externally-owned
Review Comment:
What typically owns the `PartialQueryableIndex`?
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle BEFORE submitting downloads. This sparse-allocates the bundle's container files
+ // and reserves the disk-usage accounting at bundle granularity. The returned Closeable releases the hold when
+ // the cursor closes, allowing the cache to later evict the bundle if needed.
+ // PartialBundleAcquirer.acquire() throws DruidException on failure.
+ final Closeable bundleHold = bundleAcquirer.acquire(bundleName);
+
+ try {
+ // Submit one materialization task per column so a multi-threaded download executor can fan them out
+ final ListeningExecutorService downloadExec = bundleAcquirer.getDownloadExec();
+ final List<ListenableFuture<?>> columnDownloads = new ArrayList<>(requiredColumns.size());
+ for (String column : requiredColumns) {
+ columnDownloads.add(downloadExec.submit(() -> {
+ rowSelector.getColumnHolder(column);
+ return null;
+ }));
+ }
+
+ // No canceler: downloads are allowed to complete even if the awaiter cancels. Once they complete and the inner
+ // CursorHolder is handed to asyncHolder.set(), it's owned by the asyncHolder until the awaiter takes one of
+ // two paths: asyncHolder.close() (cancel, closes the held holder) or asyncHolder.release() (success, transfers
+ // ownership to the caller). Either way the holder is drained and nothing leaks.
+ final AsyncCursorHolder asyncHolder = new AsyncCursorHolder(null);
+ Futures.addCallback(
+ Futures.allAsList(columnDownloads),
Review Comment:
Seems like a good use case for `AsyncResource.collect(columnDownloads)` and `AsyncResource.transform` (from #19539)
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle BEFORE submitting downloads. This sparse-allocates the bundle's container files
+ // and reserves the disk-usage accounting at bundle granularity. The returned Closeable releases the hold when
+ // the cursor closes, allowing the cache to later evict the bundle if needed.
+ // PartialBundleAcquirer.acquire() throws DruidException on failure.
+ final Closeable bundleHold = bundleAcquirer.acquire(bundleName);
+
+ try {
+ // Submit one materialization task per column so a multi-threaded download executor can fan them out
+ final ListeningExecutorService downloadExec = bundleAcquirer.getDownloadExec();
+ final List<ListenableFuture<?>> columnDownloads = new ArrayList<>(requiredColumns.size());
+ for (String column : requiredColumns) {
+ columnDownloads.add(downloadExec.submit(() -> {
+ rowSelector.getColumnHolder(column);
+ return null;
+ }));
+ }
+
+ // No canceler: downloads are allowed to complete even if the awaiter cancels. Once they complete and the inner
+ // CursorHolder is handed to asyncHolder.set(), it's owned by the asyncHolder until the awaiter takes one of
+ // two paths: asyncHolder.close() (cancel, closes the held holder) or asyncHolder.release() (success, transfers
+ // ownership to the caller). Either way the holder is drained and nothing leaks.
+ final AsyncCursorHolder asyncHolder = new AsyncCursorHolder(null);
+ Futures.addCallback(
+ Futures.allAsList(columnDownloads),
+ new FutureCallback<>()
+ {
+ @Override
+ public void onSuccess(List<Object> ignored)
+ {
+ try {
+ final CursorHolder inner = delegate.makeCursorHolderForProjection(spec, matched);
+ final CursorHolder holder = wrapWithBundleRelease(inner, bundleHold);
+ if (!asyncHolder.set(holder)) {
+ // wrapper was closed while we were producing the holder; close it ourselves so it doesn't leak.
+ holder.close();
+ }
+ }
+ catch (Throwable t) {
+ CloseableUtils.closeAndSuppressExceptions(bundleHold, t::addSuppressed);
+ asyncHolder.setException(t);
+ }
+ }
+
+ @Override
+ public void onFailure(Throwable t)
+ {
+ CloseableUtils.closeAndSuppressExceptions(bundleHold, t::addSuppressed);
+ asyncHolder.setException(t);
+ }
+ },
+ // Run the cursor-build callback inline on whichever download thread finishes last. The build itself does no
+ // I/O (columns are already materialized), so it doesn't need to round-trip through the download executor.
+ MoreExecutors.directExecutor()
+ );
+ return asyncHolder;
+ }
+ catch (Throwable t) {
+ // Failure between acquire and the addCallback wiring (getDownloadExec, downloadExec.submit shut-down rejection,
+ // Futures.addCallback rejecting on a bad executor, etc.). Ownership of bundleHold hasn't transferred to the
+ // callback yet, so release it here. Already-submitted download tasks will complete with no callback wired,
+ // their captured row-selector refs drop naturally.
+ throw CloseableUtils.closeAndWrapInCatch(t, bundleHold);
+ }
+ }
+
+ /**
+ * Determine the set of physical column names to required from the chosen row selector given a {@link CursorBuildSpec}
+ */
+ private static Set<String> requiredColumns(
Review Comment:
nit: the `private static` is typically placed after all non-`static` methods
##########
server/src/main/java/org/apache/druid/segment/loading/StorageLocation.java:
##########
@@ -515,6 +515,45 @@ public void release(CacheEntry entry)
}
}
+ /**
+ * Remove a {@link #weakCacheEntries} entry that currently has no outstanding holds, unlinking it from the SIEVE
+ * queue and terminating its phaser (which fires the underlying {@link CacheEntry#unmount}). No-op when the entry
+ * is absent, is a {@link #staticCacheEntries} entry, or still has outstanding holds.
+ * <p>
+ * This exists for callers that register a weak entry <em>without</em> a {@link ReservationHold} (the bootstrap
+ * reserve path uses {@link #reserveWeak}) and need to clean it up after a failed mount. The normal runtime path
Review Comment:
I think this will be useful in #19539: lingering entries on failed mount was a problem raised in review (https://github.com/apache/druid/pull/19539#discussion_r3341105641)
##########
server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:
##########
@@ -526,6 +548,345 @@ public AcquireSegmentAction acquireSegment(final DataSegment dataSegment)
}
}
+ @Override
+ public Optional<Segment> acquireCachedPartialSegment(SegmentId segmentId)
+ {
+ if (!config.isVirtualStoragePartialDownloadsEnabled()) {
+ return acquireCachedSegment(segmentId);
+ }
+ return acquireCachedInternal(segmentId, false);
+ }
+
+ @Override
+ public AcquireSegmentAction acquirePartialSegment(DataSegment dataSegment)
+ {
+ final SegmentRangeReader rangeReader = tryOpenRangeReader(dataSegment);
+ if (rangeReader == null) {
+ // Storage backend doesn't support range reads for this segment (e.g. V9 format or zipped)
+ return acquireSegment(dataSegment);
+ }
+ return acquirePartialInternal(dataSegment, rangeReader, false);
+ }
+
+
+ /**
+ * Shared implementation for {@link #acquireCachedSegment} and {@link #acquireCachedPartialSegment}. Walks the
+ * storage locations checking for existing entries.
+ * <p>
+ * When {@code requireFullyDownloaded} is {@code true} the entry must also report
+ * {@link SegmentCacheEntry#isFullyDownloaded()} (the contract for {@link #acquireCachedSegment}), which never
+ * hands back an entry that isn't behaviorally fully-materialized. When {@code false} a mounted entry is
+ * sufficient ({@link #acquireCachedPartialSegment}'s lazy-mount contract), even if some bundles are still
+ * not yet on disk. In either case {@link SegmentCacheEntry#acquireReference(Closeable)} composes the weak-entry hold
+ * into the returned segment's close lifecycle.
+ */
+ private Optional<Segment> acquireCachedInternal(SegmentId segmentId, boolean requireFullyDownloaded)
+ {
+ final SegmentCacheEntryIdentifier id = new SegmentCacheEntryIdentifier(segmentId);
+ for (StorageLocation location : locations) {
+ final SegmentCacheEntry staticEntry = location.getStaticCacheEntry(id);
+ if (staticEntry != null) {
+ if (staticEntry.isMounted() && (!requireFullyDownloaded || staticEntry.isFullyDownloaded())) {
+ return staticEntry.acquireReference();
+ }
+ return Optional.empty();
+ }
+ if (!config.isVirtualStorage()) {
+ continue;
+ }
+ final StorageLocation.ReservationHold<SegmentCacheEntry> hold = location.addWeakReservationHoldIfExists(id);
+ if (hold != null) {
+ if (hold.getEntry().isMounted() && (!requireFullyDownloaded || hold.getEntry().isFullyDownloaded())) {
+ return hold.getEntry().acquireReference(hold);
+ }
+ hold.close();
+ return Optional.empty();
+ }
+ }
+ return Optional.empty();
+ }
+
+ /**
+ * Shared scaffolding for both partial acquire APIs. {@code fullDownload=false} powers
+ * {@link #acquirePartialSegment} (lazy mount, header bytes only; bundles mount on demand at query time);
+ * {@code fullDownload=true} powers the partial-eligible branch of {@link #acquireSegment} (mount +
+ * {@link PartialSegmentFileMapperV10#ensureAllDownloaded} so the returned segment is fully-materialized).
+ * <p>
+ * Fast path: {@link #findExistingPartialWithHold} locates an existing entry across locations under a hold. If the
+ * entry is already usable (mounted, and fully-downloaded when required), return an immediate-future action whose
+ * {@code loadCleanup} is the hold (the supplier mints fresh segments per call, each with its own metadata
+ * reference, so no separate cleanup is needed).
+ * <p>
+ * Slow path: under the per-segment lock, {@link #findOrReservePartial} reuses an existing not-yet-usable entry or
+ * reserves a fresh weak one, and the action submits mount (+ optional ensureAllDownloaded) to
+ * {@link #virtualStorageLoadOnDemandExec} so callers that yield on the future never block a processing thread on
+ * deep-storage I/O.
+ */
+ private AcquireSegmentAction acquirePartialInternal(
+ DataSegment dataSegment,
+ SegmentRangeReader rangeReader,
+ boolean fullDownload
Review Comment:
On the `fullDownload` path I don't see stuff that is creating a cache entry and hold on the `StorageLocation` to account for the storage space. The logic seems to be fetching things outside of a hold. Is that right or did I miss it?
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle BEFORE submitting downloads. This sparse-allocates the bundle's container files
+ // and reserves the disk-usage accounting at bundle granularity. The returned Closeable releases the hold when
+ // the cursor closes, allowing the cache to later evict the bundle if needed.
+ // PartialBundleAcquirer.acquire() throws DruidException on failure.
+ final Closeable bundleHold = bundleAcquirer.acquire(bundleName);
+
+ try {
+ // Submit one materialization task per column so a multi-threaded download executor can fan them out
+ final ListeningExecutorService downloadExec = bundleAcquirer.getDownloadExec();
+ final List<ListenableFuture<?>> columnDownloads = new ArrayList<>(requiredColumns.size());
+ for (String column : requiredColumns) {
+ columnDownloads.add(downloadExec.submit(() -> {
+ rowSelector.getColumnHolder(column);
+ return null;
+ }));
+ }
+
+ // No canceler: downloads are allowed to complete even if the awaiter cancels. Once they complete and the inner
Review Comment:
why not cancel them? Is it because canceling them is awkward for some reason, or you feel it's better to not cancel them? I suppose there are pros and cons. The main pro I can think of (for canceling) is that no other query may need the files, so we're wasting time fetching them. The main con is that some other query _does_ need the files, so we're going to need to re-start the fetch if we cancel it.
Maybe the best behavior is to cancel it unless some other query is also waiting for the same thing.
##########
processing/src/main/java/org/apache/druid/segment/PartialQueryableIndexCursorFactory.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Order;
+import org.apache.druid.query.OrderBy;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.projections.Projections;
+import org.apache.druid.segment.projections.QueryableProjection;
+import org.apache.druid.segment.vector.VectorCursor;
+import org.apache.druid.utils.CloseableUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Partial-aware {@link CursorFactory} for {@link PartialQueryableIndex}.
+ * <p>
+ * <b>Sync vs async contract.</b> {@link #makeCursorHolder} requires the segment to already be fully downloaded,
+ * intended for callers that acquired the segment via the eager (download-everything-up-front) path, so by the time
+ * they ask for a cursor every internal file is on disk. If anything is missing it throws
+ * {@link DruidException#defensive} so that we never trigger downloads on the sync path, since processing threads must
+ * not block on deep-storage I/O. {@link #makeCursorHolderAsync} is the only path that performs downloads on demand;
+ * callers acknowledge that by opting into the async variant when they acquire a partial segment.
+ * <p>
+ * <b>Async download granularity.</b> Pre-fetch is column-level. {@link #makeCursorHolderAsync} calls
+ * {@link QueryableIndex#getColumnHolder} on each required column; the memoized supplier on the underlying
+ * {@link PartialQueryableIndex} eagerly invokes
+ * {@link org.apache.druid.segment.file.PartialSegmentFileMapperV10#mapFile} inside that call, which is what triggers
+ * the deep-storage range read. The cursor holder constructed afterward sees the already-materialized holders via the
+ * same memoized suppliers, so no further downloads happen at cursor-read time.
+ * <p>
+ * If a projection matches, the required columns are looked up against the projection's row selector and its rewritten
+ * {@link CursorBuildSpec} (which carries physical columns in the projection's namespace). When
+ * {@link CursorBuildSpec#getPhysicalColumns()} is {@code null}, every column on the chosen row selector is pre-fetched
+ * as required by the contract of {@link CursorBuildSpec}.
+ * <p>
+ * <b>Parallelism.</b> Each column's materialization is submitted as a separate task to the supplied download executor.
+ * The cursor holder is constructed once every column task has completed.
+ */
+public class PartialQueryableIndexCursorFactory implements CursorFactory
+{
+ private final PartialQueryableIndex index;
+ private final QueryableIndexCursorFactory delegate;
+ private final PartialBundleAcquirer bundleAcquirer;
+
+ public PartialQueryableIndexCursorFactory(
+ PartialQueryableIndex index,
+ TimeBoundaryInspector timeBoundaryInspector,
+ PartialBundleAcquirer bundleAcquirer
+ )
+ {
+ this.index = index;
+ this.delegate = new QueryableIndexCursorFactory(index, timeBoundaryInspector);
+ this.bundleAcquirer = bundleAcquirer;
+ }
+
+ @Override
+ public CursorHolder makeCursorHolder(CursorBuildSpec spec)
+ {
+ // refuse to download here so we never accidentally block a processing thread
+ if (!index.isFullyDownloaded()) {
+ throw DruidException.defensive(
+ "Sync makeCursorHolder requires the segment to be fully downloaded; use makeCursorHolderAsync for "
+ + "on-demand loading, or acquire the segment via the eager path so all files are loaded up front."
+ );
+ }
+ return delegate.makeCursorHolder(spec);
+ }
+
+ @Override
+ public AsyncCursorHolder makeCursorHolderAsync(CursorBuildSpec spec)
+ {
+ final QueryableProjection<QueryableIndex> matched = index.getProjection(spec);
+ final QueryableIndex rowSelector = matched != null ? matched.getRowSelector() : index;
+ final Set<String> requiredColumns = requiredColumns(rowSelector, matched, spec);
+ final String bundleName = matched != null ? matched.getName() : Projections.BASE_TABLE_PROJECTION_NAME;
+
+ // Mount the cache-layer bundle BEFORE submitting downloads. This sparse-allocates the bundle's container files
+ // and reserves the disk-usage accounting at bundle granularity. The returned Closeable releases the hold when
+ // the cursor closes, allowing the cache to later evict the bundle if needed.
+ // PartialBundleAcquirer.acquire() throws DruidException on failure.
+ final Closeable bundleHold = bundleAcquirer.acquire(bundleName);
+
+ try {
+ // Submit one materialization task per column so a multi-threaded download executor can fan them out
+ final ListeningExecutorService downloadExec = bundleAcquirer.getDownloadExec();
+ final List<ListenableFuture<?>> columnDownloads = new ArrayList<>(requiredColumns.size());
+ for (String column : requiredColumns) {
+ columnDownloads.add(downloadExec.submit(() -> {
+ rowSelector.getColumnHolder(column);
+ return null;
+ }));
+ }
+
+ // No canceler: downloads are allowed to complete even if the awaiter cancels. Once they complete and the inner
+ // CursorHolder is handed to asyncHolder.set(), it's owned by the asyncHolder until the awaiter takes one of
+ // two paths: asyncHolder.close() (cancel, closes the held holder) or asyncHolder.release() (success, transfers
+ // ownership to the caller). Either way the holder is drained and nothing leaks.
+ final AsyncCursorHolder asyncHolder = new AsyncCursorHolder(null);
+ Futures.addCallback(
+ Futures.allAsList(columnDownloads),
+ new FutureCallback<>()
+ {
+ @Override
+ public void onSuccess(List<Object> ignored)
+ {
+ try {
+ final CursorHolder inner = delegate.makeCursorHolderForProjection(spec, matched);
+ final CursorHolder holder = wrapWithBundleRelease(inner, bundleHold);
+ if (!asyncHolder.set(holder)) {
+ // wrapper was closed while we were producing the holder; close it ourselves so it doesn't leak.
+ holder.close();
+ }
+ }
+ catch (Throwable t) {
+ CloseableUtils.closeAndSuppressExceptions(bundleHold, t::addSuppressed);
+ asyncHolder.setException(t);
+ }
+ }
+
+ @Override
+ public void onFailure(Throwable t)
+ {
+ CloseableUtils.closeAndSuppressExceptions(bundleHold, t::addSuppressed);
+ asyncHolder.setException(t);
+ }
+ },
+ // Run the cursor-build callback inline on whichever download thread finishes last. The build itself does no
+ // I/O (columns are already materialized), so it doesn't need to round-trip through the download executor.
+ MoreExecutors.directExecutor()
+ );
+ return asyncHolder;
+ }
+ catch (Throwable t) {
+ // Failure between acquire and the addCallback wiring (getDownloadExec, downloadExec.submit shut-down rejection,
+ // Futures.addCallback rejecting on a bad executor, etc.). Ownership of bundleHold hasn't transferred to the
+ // callback yet, so release it here. Already-submitted download tasks will complete with no callback wired,
+ // their captured row-selector refs drop naturally.
+ throw CloseableUtils.closeAndWrapInCatch(t, bundleHold);
+ }
+ }
+
+ /**
+ * Determine the set of physical column names to required from the chosen row selector given a {@link CursorBuildSpec}
+ */
+ private static Set<String> requiredColumns(
+ QueryableIndex rowSelector,
+ @Nullable QueryableProjection<QueryableIndex> matched,
+ CursorBuildSpec originalSpec
+ )
+ {
+ final CursorBuildSpec effective = matched != null ? matched.getCursorBuildSpec() : originalSpec;
+ if (effective.getPhysicalColumns() != null) {
+ final Set<String> required = new LinkedHashSet<>(effective.getPhysicalColumns());
+ // physicalColumns enumerates the SELECTED columns, but QueryableIndexCursorHolder also reads __time while
+ // building the cursor, independent of physicalColumns: unconditionally for a time-ordered index (its
+ // interval-checking offset reads timestamps), and via a synthesized __time range filter for a non-time-ordered
+ // index whose data extends past the query interval. That read happens after the cursor holder is handed back,
+ // so __time must be pre-fetched on the async path or it becomes a lazy deep-storage download on a processing
+ // thread. Predicting exactly when the holder reads __time would mean replicating its internals (fragile, and it
+ // reads __time in the common cases anyway), so always include it: __time is cheap, and it resolves to a
+ // no-download constant column for projections that don't carry a real time column.
+ required.add(ColumnHolder.TIME_COLUMN_NAME);
+ return required;
+ }
+ // Conservative fallback when physicalColumns isn't declared, fetch every column on the chosen row selector
+ // plus __time (which is special-cased and not enumerated by getColumnNames()).
Review Comment:
Seems weird. Can you consider changing it in this PR, so `getColumnNames()` does include `__time`? Looking around at the usages of `getColumnNames()`, it doesn't seem like anything is super-dependent on them not including `__time`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org