You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "uds5501 (via GitHub)" <gi...@apache.org> on 2025/09/08 13:54:19 UTC
[PR] Emit event when a datasource has been dropped from the coordinator (druid)
uds5501 opened a new pull request, #18497:
URL: https://github.com/apache/druid/pull/18497
Follow up PR for https://github.com/apache/druid/pull/18495
It introduces a new event `segment/schemaCache/datasource/dropped/count` that is emitted when the data source is dropped from the coordinator and the cache refresh in broker acknowledges it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Emit event when a datasource has been dropped from the coordinator (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18497:
URL: https://github.com/apache/druid/pull/18497#discussion_r2330481341
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCache.java:
##########
@@ -243,13 +246,14 @@ public void refresh(final Set<SegmentId> segmentsToRefresh, final Set<String> da
dataSourcesNeedingRebuild.clear();
}
-
+ List<String> droppedDataSources = new ArrayList<>();
// Rebuild the datasources.
for (String dataSource : dataSourcesToRebuild) {
final RowSignature rowSignature = buildDataSourceRowSignature(dataSource);
if (rowSignature == null) {
log.info("datasource [%s] no longer exists, all metadata removed.", dataSource);
tables.remove(dataSource);
+ droppedDataSources.add(dataSource);
Review Comment:
Rather than adding to the list, let's just emit the event here.
##########
server/src/main/java/org/apache/druid/segment/metadata/Metric.java:
##########
@@ -49,6 +49,7 @@ public class Metric
public static final String STARTUP_DURATION_MILLIS = "metadatacache/init/time";
public static final String REFRESHED_SEGMENTS = PREFIX + "refresh/count";
public static final String REFRESH_DURATION_MILLIS = PREFIX + "refresh/time";
+ public static final String DATASOURCE_DROPPED = PREFIX + "datasource/dropped/count";
Review Comment:
```suggestion
public static final String DATASOURCE_REMOVED = PREFIX + "dataSource/removed";
```
##########
docs/operations/metrics.md:
##########
@@ -441,6 +441,7 @@ These metrics are emitted by the Druid Coordinator in every run of the correspon
|`serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator has been failing to sync with a segment-loading server. Emitted only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled.|`server`, `tier`|Not emitted for synced servers.|
|`metadatacache/init/time`|Time taken to initialize the coordinator segment metadata cache.||Depends on the number of segments.|
|`segment/schemaCache/refresh/count`|Number of segments for which schema was refreshed in coordinator segment schema cache.|`dataSource`||
+|`segment/schemaCache/datasource/dropped/count`|Number of datasources dropped from broker cache after segments were marked unused.|`dataSource`||
Review Comment:
```suggestion
|`segment/schemaCache/dataSource/removed`|Emitted when a datasource is removed from the Broker cache due to segments being marked as unused.|`dataSource`||
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Emit event when a datasource has been dropped from the coordinator (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe merged PR #18497:
URL: https://github.com/apache/druid/pull/18497
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Emit event when a datasource has been dropped from the coordinator (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18497:
URL: https://github.com/apache/druid/pull/18497#discussion_r2332191950
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.java:
##########
@@ -177,6 +177,13 @@ public void test_runIndexTask_andKillData()
.hasService("druid/coordinator"),
agg -> agg.hasSumAtLeast(numSegments)
);
+
+ eventCollector.latchableEmitter().waitForEvent(
Review Comment:
```suggestion
// Wait for the Broker to remove this datasource from its schema cache
eventCollector.latchableEmitter().waitForEvent(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Emit event when a datasource has been dropped from the coordinator (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18497:
URL: https://github.com/apache/druid/pull/18497#discussion_r2332186529
##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedClusterApis.java:
##########
@@ -296,6 +296,10 @@ public void waitForAllSegmentsToBeAvailable(String dataSource, EmbeddedCoordinat
.hasDimension(DruidMetrics.DATASOURCE, dataSource),
agg -> agg.hasSumAtLeast(numSegments)
);
+ broker.latchableEmitter().waitForEvent(
Review Comment:
Maybe we should we look for aggregate where the sum is equal to numSegments.
After this change, do we even need to wait to wait for the load success.
@uds5501 , could you try to push a change with the coordinator part commented out?
If not needed, we can remove that check completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Emit event when a datasource has been dropped from the coordinator (druid)
Posted by "uds5501 (via GitHub)" <gi...@apache.org>.
uds5501 commented on PR #18497:
URL: https://github.com/apache/druid/pull/18497#issuecomment-3268824382
@kfaraz yeah let me do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Emit event when a datasource has been dropped from the coordinator (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18497:
URL: https://github.com/apache/druid/pull/18497#discussion_r2332187806
##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedClusterApis.java:
##########
@@ -284,7 +284,7 @@ public void verifyNumVisibleSegmentsIs(int numExpectedSegments, String dataSourc
* Waits for all used segments (including overshadowed) of the given datasource
* to be loaded on historicals.
Review Comment:
```suggestion
* Waits for all used segments (including overshadowed) of the given datasource
* to be queryable by Brokers.
```
##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedClusterApis.java:
##########
@@ -284,7 +284,7 @@ public void verifyNumVisibleSegmentsIs(int numExpectedSegments, String dataSourc
* Waits for all used segments (including overshadowed) of the given datasource
* to be loaded on historicals.
Review Comment:
```suggestion
* Waits for all used segments (including overshadowed) of the given datasource
* to be queryable by the given Broker.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Emit event when a datasource has been dropped from the coordinator (druid)
Posted by "uds5501 (via GitHub)" <gi...@apache.org>.
uds5501 commented on code in PR #18497:
URL: https://github.com/apache/druid/pull/18497#discussion_r2332250993
##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedClusterApis.java:
##########
@@ -296,6 +296,10 @@ public void waitForAllSegmentsToBeAvailable(String dataSource, EmbeddedCoordinat
.hasDimension(DruidMetrics.DATASOURCE, dataSource),
agg -> agg.hasSumAtLeast(numSegments)
);
+ broker.latchableEmitter().waitForEvent(
Review Comment:
Tested locally, the tests seems to be working fine without the coordinator wait, removed in the latest comment.
Also updated the EmbeddedBroker as part of the numSegments change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Emit event when a datasource has been dropped from the coordinator (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on PR #18497:
URL: https://github.com/apache/druid/pull/18497#issuecomment-3271501617
One input source IT was running. The patch does not change anything there hence merged the PR since its a dev blocker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org