You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "GWphua (via GitHub)" <gi...@apache.org> on 2026/03/09 10:30:01 UTC
[I] MSQ Compaction Not Working due to Metadata Cache Sync (druid)
GWphua opened a new issue, #19112:
URL: https://github.com/apache/druid/issues/19112
Affected Version
Master Branch
Between v36~v37: (HEAD at commit 8307d8a26c, PR#19030)
Encountered when trying out MSQ Compaction using MM-less ingestion.
The test datasource is ingested via Kafka supervisor.
Relevant Overlord settings:
```
## Required for Supervisor compaction in v36
druid_supervisor_compaction_enabled: true
druid_supervisor_compaction_engine: "msq"
druid_manager_segments_useIncrementalCache: "always"
// Loadlist for MSQ extension + MM-less
```
Description
I have set up supervisor auto-compaction, using [this](https://druid.apache.org/docs/latest/api-reference/automatic-compaction-api#update-cluster-level-compaction-config) and created a supervisor to compact my datasource.
However, the supervisor keeps showing 'RUNNING', with no tasks being created. Looking at the overlord logs, it seems like there's some problem metadata cache.
org.apache.druid.metadata.segment.SqlSegmentMetadataTransactionFactory - Starting read-write transaction for datasource[test_ds]. Reads will be done directly from metadata store since cache is not synced yet.
I am testing this on a test cluster, with only 1 datasource of 6065 segments. The logs says that it is trying to sync the cache for 2 hours already. I'm not sure if there's some kind of deadlocks preventing the sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] MSQ Compaction Not Working on MM-less (Metadata Cache Syncing Forever) (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on issue #19112:
URL: https://github.com/apache/druid/issues/19112#issuecomment-4029303004
@kfaraz
I found the fix to the problem.
It turns out that in my cluster configuration, the following config is only provided for the Coordinator. The Overlord did not have this config.
```
druid_coordinator_asOverlord_enabled: false
```
Failure to configure this config on the Overlord is harmless before #17935. The fact that the config is prefixed with `druid_coordinator`, and there is no mention in the documentations, made me believe that there is no need to consider applying this configuration on the Overlord.
After #17935, if users fail to add this config, `isStandalone()` will return false. Users will keep seeing `Waiting for cache to finish sync with metadata store.` in the logs, but the sync will never finish, because the Metadata Cache Sync will is not started at all.
```
if (isStandalone()) {
segmentMetadataCache.start();
}
// How isStandalone is resolved.
private boolean isStandalone()
{
return !coordinatorOverlordServiceConfig.isEnabled();
}
```
Documentation / upgrade notes will be helpful for users looking to upgrade / use this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] MSQ Compaction Not Working (Overlord Metadata Cache Syncing Forever) (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on issue #19112:
URL: https://github.com/apache/druid/issues/19112#issuecomment-4029365789
I think it's because we are simply copying the configs from `examples/conf/druid/auto/coordinator-overlord/runtime.properties`
In here, `druid.coordinator.asOverlord.enabled=true` and hence the default is being overwritten.
Then, by specifying `druid_coordinator_asOverlord_enabled: false` in our Helm chart, we overwrite this again back to false 🤣
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] MSQ Compaction Not Working on MM-less (Metadata Cache Syncing Forever) (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on issue #19112:
URL: https://github.com/apache/druid/issues/19112#issuecomment-4028501731
Hi @kfaraz
I have met an NPE bug while looking at the problem. Proposed fix is in #19118.
I have also tried setting the configs as you advised. However, I think something might be wrong with the metadata cache sync. For a cluster of 6000 segments, this should not take a long time.
```
2026-03-10T03:38:59,523 WARN [qtp1741034833-58] org.apache.druid.metadata.segment.SqlSegmentMetadataTransactionFactory - Starting read-write transaction for datasource[pingmesh.inter_idc_rtt_projected]. Reads will be done directly from metadata store since cache is not synced yet.
2026-03-10T03:38:59,523 INFO [qtp1741034833-58] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Waiting for cache to finish sync with metadata store.
2026-03-10T03:39:02,222 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Notified task runner to clean up [0] tasks with IDs[[]].
2026-03-10T03:39:02,230 INFO [TaskQueue-StorageSync] org.apache.druid.indexing.overlord.TaskQueue - Synced [0] tasks from storage (0 tasks added, 0 tasks removable, 0 tasks removed).
2026-03-10T03:39:02,323 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Notified task runner to clean up [0] tasks with IDs[[]].
2026-03-10T03:40:02,220 INFO [TaskQueue-StorageSync] org.apache.druid.indexing.overlord.TaskQueue - Synced [0] tasks from storage (0 tasks added, 0 tasks removable, 0 tasks removed).
2026-03-10T03:40:02,221 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Notified task runner to clean up [0] tasks with IDs[[]].
2026-03-10T03:41:00,024 WARN [qtp1741034833-72] org.apache.druid.metadata.segment.SqlSegmentMetadataTransactionFactory - Starting read-write transaction for datasource[pingmesh.inter_idc_rtt_projected]. Reads will be done directly from metadata store since cache is not synced yet.
2026-03-10T03:41:00,024 INFO [qtp1741034833-72] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Waiting for cache to finish sync with metadata store.
... // 7 minutes of related logs
2026-03-10T03:48:59,523 INFO [qtp1741034833-58] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Wait complete. Cache is now in state[LEADER_FIRST_SYNC_PENDING].
2026-03-10T03:48:59,524 INFO [qtp1741034833-58] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Waiting for cache to finish sync with metadata store.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] MSQ Compaction Not Working on MM-less (Metadata Cache Syncing Forever) (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on issue #19112:
URL: https://github.com/apache/druid/issues/19112#issuecomment-4028791544
@GWphua , I have approved #19118 , thanks for the fix!
The cache taking too long to sync 6000 segments is indeed weird. It shouldn't take more than a couple of seconds.
This code has not been touched recently either and I have been using it in production for quite some time now.
Is the cache able to sync even once? How about the Coordinator, is that also taking time to sync?
You can set the property `druid.manager.segments.useIncrementalCache: always` on Coordinator too to see if the cache works fine there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] MSQ Compaction Not Working (Overlord Metadata Cache Syncing Forever) (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on issue #19112:
URL: https://github.com/apache/druid/issues/19112#issuecomment-4029424938
I feel that some docs will still help, especially maybe a line saying that the `druid.coordinator.asOverlord.enabled=true` is actually needed on the overlord for other operators to be aware of this config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] MSQ Compaction Not Working on MM-less (Metadata Cache Syncing Forever) (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on issue #19112:
URL: https://github.com/apache/druid/issues/19112#issuecomment-4023147284
Thanks for reporting the issue, @GWphua !
It seems that the docs are out-of-date. The property `druid.supervisor.compaction.enabled` and `druid.supervisor.compaction.engine` are not used anymore.
Instead, these fields are now available in the compaction dynamic config.
Set `useSupervisors` to `true` and `engine` to `msq` for your use case.
Fix for docs: https://github.com/apache/druid/pull/19113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] MSQ Compaction Not Working (Overlord Metadata Cache Syncing Forever) (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on issue #19112:
URL: https://github.com/apache/druid/issues/19112#issuecomment-4029407160
> Understand that https://github.com/apache/druid/pull/17935 is introduced a while back. If there's a need to propagate the docs back to earlier versions (if applicable), maybe a new PR is better to do backports (assuming we plan to)?
Given that the issue was the config setup, I think we may just skip the docs.
The default behaviour of the config (`false`) is as expected and in that case, the Overlord cache would work fine.
Setting `druid.coordinator.asOverlord.enabled=true` on the Overlord when it is actually intended to run in a standalone mode is actually a misconfiguration anyway.
Let me know if you still feel that some docs would be helpful for this. Or if you have the changes in mind, please feel free to create a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] MSQ Compaction Not Working (Overlord Metadata Cache Syncing Forever) (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on issue #19112:
URL: https://github.com/apache/druid/issues/19112#issuecomment-4029378878
> > Documentation / upgrade notes will be helpful for users looking to upgrade / use this feature.
>
> Fair enough. I already have a PR up, let me include this there.
Understand that #17935 is introduced a while back. If there's a need to propagate the docs back to earlier versions, maybe a new PR is better to do backports (assuming we plan to)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] MSQ Compaction Not Working (Overlord Metadata Cache Syncing Forever) (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on issue #19112:
URL: https://github.com/apache/druid/issues/19112#issuecomment-4029340304
Oh, so you had to explicitly set `druid.coordinator.asOverlord.enabled: false`? But this config is already false by default.
Was the Overlord getting properties from some other source/template as well, which was probably setting this to true first?
> Documentation / upgrade notes will be helpful for users looking to upgrade / use this feature.
Fair enough. I already have a PR up, let me include this there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org