You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2026/07/13 23:51:29 UTC
[PR] feat: partial load rule announcement (druid)
clintropolis opened a new pull request, #19682:
URL: https://github.com/apache/druid/pull/19682
### Description
Follow-up to #19671, adds the wiring for partial loaded segments from partial load rules to announce the partially loaded size instead of the full segment size, so that the coordinator side accounting can more accurately manage cache placement. The main change is that`SegmentCacheManager` load and bootstrap methods now return a `DataSegment` which implementations can use to thread a `DataSegmentAndLoadProfile` to the announcer to do the partial announcement (same type used coordinator side to carry non-serializing 'side-car' information about partial load stuff)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: partial load rule announcement (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis merged PR #19682:
URL: https://github.com/apache/druid/pull/19682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: partial load rule announcement (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19682:
URL: https://github.com/apache/druid/pull/19682#discussion_r3583764322
##########
server/src/test/java/org/apache/druid/segment/loading/SegmentLocalCacheManagerPartialRuleLoadTest.java:
##########
@@ -289,6 +291,101 @@ void testLoadEagerlyDownloadsBaseDependencyOfSelectedProjection() throws Excepti
);
}
+ @Test
+ void testLoadPartialReturnsDataSegmentAndLoadProfileWithRealizedBytes() throws Exception
+ {
+ // load() on the partial-load path returns a DataSegmentAndLoadProfile carrying a PartialLoadProfile.forLoaded with
+ // the actual on-disk footprint. Under a projection rule, that footprint is metadata header + rule-selected
+ // projection bundle + __base (transitive dependency pinned via the projection's parent-hold). Anything less
+ // would under-report to the coordinator: the historical downloads __base as part of the rule (see
+ // awaitEagerDownloadsOrClearRule's dep expansion), so its bytes belong in the announcement's loadedBytes.
+ manager = makeManager(true, true);
+ final StorageLocation location = manager.getLocations().get(0);
+
+ final DataSegment loaded = manager.load(partialWrapperSegment(List.of(AGG_BUNDLE)));
+
+ Assertions.assertInstanceOf(
+ DataSegmentAndLoadProfile.class,
+ loaded,
+ "load on the partial-rule path must return a DataSegmentAndLoadProfile so the announcer can report actual footprint"
+ );
+ final PartialLoadProfile profile = ((DataSegmentAndLoadProfile) loaded).profile();
+ Assertions.assertEquals(FINGERPRINT, profile.fingerprint(), "profile's fingerprint must match the applied rule");
+
+ // Sum the parts we expect: metadata + AGG_BUNDLE (rule-selected) + __base (transitive dep).
+ final PartialSegmentMetadataCacheEntry metadata = weakReservedMetadata(location, SEGMENT_ID);
+ final PartialSegmentBundleCacheEntry aggEntry = location.getCacheEntry(
+ new PartialSegmentBundleCacheEntryIdentifier(SEGMENT_ID, AGG_BUNDLE)
+ );
+ final PartialSegmentBundleCacheEntry baseEntry = location.getCacheEntry(
+ new PartialSegmentBundleCacheEntryIdentifier(SEGMENT_ID, Projections.BASE_TABLE_PROJECTION_NAME)
+ );
+ Assertions.assertNotNull(aggEntry, "rule-selected projection bundle should be present");
+ Assertions.assertNotNull(baseEntry, "transitive __base dependency should be present");
+ final long expectedRealizedBytes = metadata.getSize() + aggEntry.getSize() + baseEntry.getSize();
+
+ Assertions.assertEquals(
+ Long.valueOf(expectedRealizedBytes),
+ profile.loadedBytes(),
+ "profile's loadedBytes must include metadata + rule-selected projection + __base dependency; missing any of "
+ + "these would under-report the on-disk footprint to the coordinator"
+ );
+ Assertions.assertTrue(
+ baseEntry.getSize() > 0,
+ "sanity check: __base's size should be non-zero (so the dep-inclusion assertion is meaningful)"
+ );
+ }
+ @Test
+ void testRealizedBytesIncludesPinnedBaseDependency() throws Exception
+ {
+ manager = makeManager(true, true);
+ final StorageLocation location = manager.getLocations().get(0);
+
+ manager.load(partialWrapperSegment(List.of(AGG_BUNDLE)));
+
+ final PartialSegmentMetadataCacheEntry metadata = weakReservedMetadata(location, SEGMENT_ID);
+ final long metadataBytes = metadata.getSize();
+ final long aggBytes = bundleEntry(location, AGG_BUNDLE).getSize();
+ final long baseBytes = bundleEntry(location, Projections.BASE_TABLE_PROJECTION_NAME).getSize();
+
+ Assertions.assertTrue(baseBytes > 0, "base dependency must occupy real on-disk bytes");
+
+ final long realized = metadata.getRealizedBytes();
+
+ Assertions.assertEquals(
+ metadataBytes + aggBytes + baseBytes,
+ realized,
+ String.format(
Review Comment:
heh no worries, will fix up shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: partial load rule announcement (druid)
Posted by "capistrant (via GitHub)" <gi...@apache.org>.
capistrant commented on code in PR #19682:
URL: https://github.com/apache/druid/pull/19682#discussion_r3583726098
##########
server/src/test/java/org/apache/druid/segment/loading/SegmentLocalCacheManagerPartialRuleLoadTest.java:
##########
@@ -289,6 +291,101 @@ void testLoadEagerlyDownloadsBaseDependencyOfSelectedProjection() throws Excepti
);
}
+ @Test
+ void testLoadPartialReturnsDataSegmentAndLoadProfileWithRealizedBytes() throws Exception
+ {
+ // load() on the partial-load path returns a DataSegmentAndLoadProfile carrying a PartialLoadProfile.forLoaded with
+ // the actual on-disk footprint. Under a projection rule, that footprint is metadata header + rule-selected
+ // projection bundle + __base (transitive dependency pinned via the projection's parent-hold). Anything less
+ // would under-report to the coordinator: the historical downloads __base as part of the rule (see
+ // awaitEagerDownloadsOrClearRule's dep expansion), so its bytes belong in the announcement's loadedBytes.
+ manager = makeManager(true, true);
+ final StorageLocation location = manager.getLocations().get(0);
+
+ final DataSegment loaded = manager.load(partialWrapperSegment(List.of(AGG_BUNDLE)));
+
+ Assertions.assertInstanceOf(
+ DataSegmentAndLoadProfile.class,
+ loaded,
+ "load on the partial-rule path must return a DataSegmentAndLoadProfile so the announcer can report actual footprint"
+ );
+ final PartialLoadProfile profile = ((DataSegmentAndLoadProfile) loaded).profile();
+ Assertions.assertEquals(FINGERPRINT, profile.fingerprint(), "profile's fingerprint must match the applied rule");
+
+ // Sum the parts we expect: metadata + AGG_BUNDLE (rule-selected) + __base (transitive dep).
+ final PartialSegmentMetadataCacheEntry metadata = weakReservedMetadata(location, SEGMENT_ID);
+ final PartialSegmentBundleCacheEntry aggEntry = location.getCacheEntry(
+ new PartialSegmentBundleCacheEntryIdentifier(SEGMENT_ID, AGG_BUNDLE)
+ );
+ final PartialSegmentBundleCacheEntry baseEntry = location.getCacheEntry(
+ new PartialSegmentBundleCacheEntryIdentifier(SEGMENT_ID, Projections.BASE_TABLE_PROJECTION_NAME)
+ );
+ Assertions.assertNotNull(aggEntry, "rule-selected projection bundle should be present");
+ Assertions.assertNotNull(baseEntry, "transitive __base dependency should be present");
+ final long expectedRealizedBytes = metadata.getSize() + aggEntry.getSize() + baseEntry.getSize();
+
+ Assertions.assertEquals(
+ Long.valueOf(expectedRealizedBytes),
+ profile.loadedBytes(),
+ "profile's loadedBytes must include metadata + rule-selected projection + __base dependency; missing any of "
+ + "these would under-report the on-disk footprint to the coordinator"
+ );
+ Assertions.assertTrue(
+ baseEntry.getSize() > 0,
+ "sanity check: __base's size should be non-zero (so the dep-inclusion assertion is meaningful)"
+ );
+ }
+ @Test
+ void testRealizedBytesIncludesPinnedBaseDependency() throws Exception
+ {
+ manager = makeManager(true, true);
+ final StorageLocation location = manager.getLocations().get(0);
+
+ manager.load(partialWrapperSegment(List.of(AGG_BUNDLE)));
+
+ final PartialSegmentMetadataCacheEntry metadata = weakReservedMetadata(location, SEGMENT_ID);
+ final long metadataBytes = metadata.getSize();
+ final long aggBytes = bundleEntry(location, AGG_BUNDLE).getSize();
+ final long baseBytes = bundleEntry(location, Projections.BASE_TABLE_PROJECTION_NAME).getSize();
+
+ Assertions.assertTrue(baseBytes > 0, "base dependency must occupy real on-disk bytes");
+
+ final long realized = metadata.getRealizedBytes();
+
+ Assertions.assertEquals(
+ metadataBytes + aggBytes + baseBytes,
+ realized,
+ String.format(
Review Comment:
Sorry for handing you tests with forbidden api violations ðŸ«
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: partial load rule announcement (druid)
Posted by "capistrant (via GitHub)" <gi...@apache.org>.
capistrant commented on code in PR #19682:
URL: https://github.com/apache/druid/pull/19682#discussion_r3579659069
##########
server/src/main/java/org/apache/druid/server/SegmentManager.java:
##########
@@ -298,19 +306,27 @@ public void loadSegmentOnBootstrap(
*
* @param dataSegment segment to load
*
+ * @return the input {@code dataSegment}, or a
+ * {@link org.apache.druid.client.DataSegmentAndLoadProfile} wrapping it when the historical actually
+ * materialized a partial-load footprint. Callers pass the returned value to the announcement layer so
+ * partial-load announcements carry accurate {@code loadedBytes}.
* @throws SegmentLoadingException if the segment cannot be loaded
* @throws IOException if the segment info cannot be cached on disk
*/
- public void loadSegment(final DataSegment dataSegment) throws SegmentLoadingException, IOException
+ public DataSegment loadSegment(final DataSegment dataSegment) throws SegmentLoadingException, IOException
Review Comment:
+1
##########
server/src/main/java/org/apache/druid/segment/loading/PartialSegmentMetadataCacheEntry.java:
##########
@@ -620,6 +620,31 @@ public long getSize()
}
}
+ /**
+ * The rule-declared on-disk footprint for this segment: this metadata entry's own reservation (V10 header plus
+ * post-mount adjustment) plus the sum of every rule-held bundle's reservation. Deliberately excludes bundles that
+ * happen to be linked for other reasons (e.g. load on demand). Used by the historical to stamp accurate
+ * {@code loadedBytes} on partial-load announcements, so inventory-view accounting reflects what the rule was
+ * configured to pin rather than incidental cache residency.
+ */
+ public long getRealizedBytes()
+ {
+ // Read sizes straight from ruleBundleHolds under entryLock: each hold owns a final reference to its bundle entry
+ // and PartialSegmentBundleCacheEntry.getSize() reads a final long, no lock nesting, no null case, no reliance
+ // on external segment-lock ordering.
+ entryLock.lock();
+ try {
+ long total = currentSize;
+ for (StorageLocation.ReservationHold<PartialSegmentBundleCacheEntry> hold : ruleBundleHolds.values()) {
Review Comment:
hmmm. I think this sounds legit. couple of claude driven UTs for `SegmentLocalCacheManagerPartialRuleLoadTest` that could help validate any fix
```JAVA
@Test
void testRealizedBytesIncludesPinnedBaseDependency() throws Exception
{
manager = makeManager(true, true);
final StorageLocation location = manager.getLocations().get(0);
manager.load(partialWrapperSegment(List.of(AGG_BUNDLE)));
final PartialSegmentMetadataCacheEntry metadata = weakReservedMetadata(location, SEGMENT_ID);
final long metadataBytes = metadata.getSize();
final long aggBytes = bundleEntry(location, AGG_BUNDLE).getSize();
final long baseBytes = bundleEntry(location, Projections.BASE_TABLE_PROJECTION_NAME).getSize();
Assertions.assertTrue(baseBytes > 0, "base dependency must occupy real on-disk bytes");
final long realized = metadata.getRealizedBytes();
Assertions.assertEquals(
metadataBytes + aggBytes + baseBytes,
realized,
String.format(
"realizedBytes must include the pinned __base dependency: metadata=%d + selected=%d + base=%d = %d, "
+ "but got %d (short by %d, exactly the base footprint)",
metadataBytes, aggBytes, baseBytes, metadataBytes + aggBytes + baseBytes,
realized, (metadataBytes + aggBytes + baseBytes) - realized
)
);
}
@Test
void testRealizedBytesCountsSharedBaseDependencyOnce() throws Exception
{
// Two projections in a single rule both pin the same __base. The base bytes must be counted exactly once.
manager = makeManager(true, true);
final StorageLocation location = manager.getLocations().get(0);
manager.load(partialWrapperSegment(List.of(AGG_BUNDLE, OTHER_AGG_BUNDLE)));
final PartialSegmentMetadataCacheEntry metadata = weakReservedMetadata(location, SEGMENT_ID);
final long metadataBytes = metadata.getSize();
final long aggBytes = bundleEntry(location, AGG_BUNDLE).getSize();
final long otherBytes = bundleEntry(location, OTHER_AGG_BUNDLE).getSize();
final long baseBytes = bundleEntry(location, Projections.BASE_TABLE_PROJECTION_NAME).getSize();
Assertions.assertEquals(
metadataBytes + aggBytes + otherBytes + baseBytes,
metadata.getRealizedBytes(),
"realizedBytes must include the shared __base exactly once across both dependent projections"
);
}
private static CacheEntry bundleEntry(StorageLocation location, String bundleName)
{
final CacheEntry entry =
location.getCacheEntry(new PartialSegmentBundleCacheEntryIdentifier(SEGMENT_ID, bundleName));
Assertions.assertNotNull(entry, "bundle[" + bundleName + "] must be resident after load");
return entry;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: partial load rule announcement (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19682:
URL: https://github.com/apache/druid/pull/19682#discussion_r3575187012
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/query/PartialProjectionLoadRuleQueryTest.java:
##########
@@ -248,6 +255,43 @@
);
}
+ @Test
+ void testCoordinatorSeesPartialLoadFootprint()
+ {
+ // End-to-end check that PartialLoadedDataSegment reaches the coordinator's inventory accounting: the historical's
+ // announcement stamps realizedBytes (metadata + selected projection + __base dep) as loadedBytes, so
+ // sys.servers.curr_size for the historical reflects the partial footprint — strictly less than the segment's full
+ // size (which includes the unmatched projection's bytes). Without the fix, forAnnouncement would stamp
+ // segment.getSize() and curr_size would equal the full size.
+ final long fullSize = Long.parseLong(
+ cluster.callApi().runSql(
+ "SELECT \"size\" FROM sys.segments WHERE datasource = '" + dataSource + "'"
+ ).trim()
+ );
Review Comment:
## CodeQL / Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/druid/security/code-scanning/11333)
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/query/PartialProjectionLoadRuleQueryTest.java:
##########
@@ -248,6 +255,43 @@
);
}
+ @Test
+ void testCoordinatorSeesPartialLoadFootprint()
+ {
+ // End-to-end check that PartialLoadedDataSegment reaches the coordinator's inventory accounting: the historical's
+ // announcement stamps realizedBytes (metadata + selected projection + __base dep) as loadedBytes, so
+ // sys.servers.curr_size for the historical reflects the partial footprint — strictly less than the segment's full
+ // size (which includes the unmatched projection's bytes). Without the fix, forAnnouncement would stamp
+ // segment.getSize() and curr_size would equal the full size.
+ final long fullSize = Long.parseLong(
+ cluster.callApi().runSql(
+ "SELECT \"size\" FROM sys.segments WHERE datasource = '" + dataSource + "'"
+ ).trim()
+ );
+ Assertions.assertTrue(fullSize > 0, "sys.segments.size must be populated for the ingested segment");
+
+ final long currSize = Long.parseLong(
+ cluster.callApi().runSql(
+ "SELECT curr_size FROM sys.servers WHERE server_type = 'historical'"
+ ).trim()
+ );
Review Comment:
## CodeQL / Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/druid/security/code-scanning/11334)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: partial load rule announcement (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19682:
URL: https://github.com/apache/druid/pull/19682#discussion_r3578873826
##########
server/src/main/java/org/apache/druid/segment/loading/PartialSegmentMetadataCacheEntry.java:
##########
@@ -620,6 +620,31 @@ public long getSize()
}
}
+ /**
+ * The rule-declared on-disk footprint for this segment: this metadata entry's own reservation (V10 header plus
+ * post-mount adjustment) plus the sum of every rule-held bundle's reservation. Deliberately excludes bundles that
+ * happen to be linked for other reasons (e.g. load on demand). Used by the historical to stamp accurate
+ * {@code loadedBytes} on partial-load announcements, so inventory-view accounting reflects what the rule was
+ * configured to pin rather than incidental cache residency.
+ */
+ public long getRealizedBytes()
+ {
+ // Read sizes straight from ruleBundleHolds under entryLock: each hold owns a final reference to its bundle entry
+ // and PartialSegmentBundleCacheEntry.getSize() reads a final long, no lock nesting, no null case, no reliance
+ // on external segment-lock ordering.
+ entryLock.lock();
+ try {
+ long total = currentSize;
+ for (StorageLocation.ReservationHold<PartialSegmentBundleCacheEntry> hold : ruleBundleHolds.values()) {
Review Comment:
[P1] Count dependency bundles in loadedBytes
`ruleBundleHolds` contains only explicitly selected bundle names, while projection bundles indirectly pin dependencies such as `__base`. Since a bundle's `getSize()` excludes dependency sizes, `getRealizedBytes()` underreports the actual pinned footprint and coordinator capacity accounting overestimates available space. Sum the unique dependency closure as well and test metadata plus selected-bundle plus dependency bytes.
##########
server/src/main/java/org/apache/druid/server/SegmentManager.java:
##########
@@ -298,19 +306,27 @@ public void loadSegmentOnBootstrap(
*
* @param dataSegment segment to load
*
+ * @return the input {@code dataSegment}, or a
+ * {@link org.apache.druid.client.DataSegmentAndLoadProfile} wrapping it when the historical actually
+ * materialized a partial-load footprint. Callers pass the returned value to the announcement layer so
+ * partial-load announcements carry accurate {@code loadedBytes}.
* @throws SegmentLoadingException if the segment cannot be loaded
* @throws IOException if the segment info cannot be cached on disk
*/
- public void loadSegment(final DataSegment dataSegment) throws SegmentLoadingException, IOException
+ public DataSegment loadSegment(final DataSegment dataSegment) throws SegmentLoadingException, IOException
Review Comment:
[P1] Update mocks for the non-void load result
Changing `loadSegment()` to return `DataSegment` leaves `SegmentLoadDropHandlerTest` using `Mockito.doNothing()` in two setups. Mockito rejects `doNothing()` on non-void methods, so those tests fail during setup. Stub the method to return its input segment, including after the chained first-call exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org