You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "TessaIO (via GitHub)" <gi...@apache.org> on 2025/08/18 11:54:52 UTC
[I] Communication with Coordinator is broken after upgrading to 34.0.0 (druid)
TessaIO opened a new issue, #18411:
URL: https://github.com/apache/druid/issues/18411
### Affected Version
34.0.0
### Description
After upgrading one of our Druid test clusters to 34.0.0, we found that all components (routers, brokers, historicals), failed to communicate with the coordinator, and the whole cluster was down. We see the following error on these components, while there are no issues with the coordinator logs:
```
2025-08-18T11:50:57,208 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (9 of 9) in 50,867ms.
org.apache.druid.java.util.common.IOE: Retries exhausted, couldn't fulfill request to [http://druid-sb4-coordinators-76dcdd9955-x8z8q:8088/druid-ext/basic-security/authorization/db/BasicMetadataAuthor
at org.apache.druid.discovery.DruidLeaderClient.go(DruidLeaderClient.java:219) ~[druid-server-34.0.0.jar:34.0.0]
at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchGroupMappingMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:4
at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchGroupAndRoleMapFromCoordinator$6(CoordinatorPollingBasicAuthorizerCacheManager.j
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-34.0.0.jar:34.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-34.0.0.jar:34.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-34.0.0.jar:34.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-34.0.0.jar:34.0.0]
at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchGroupAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:367)
at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$1(CoordinatorPollingBasicAuthorizerCacheManager.java:159) ~[?:?]
at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:56) [druid-processing-34.0.0.jar:34.0.0]
at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:52) [druid-processing-34.0.0.jar:34.0.0]
at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:88) [druid-processing-34.0.0.jar:34.0.0]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
```
It looks like it's related to the basic auth configuration. This is the Auth Config:
```
druid.auth.authenticatorChain=["BasicMetadataAuthenticator"]
druid.auth.authenticator.BasicMetadataAuthenticator.type=basic
# Default password for 'admin' user
druid.auth.authenticator.BasicMetadataAuthenticator.initialAdminPassword=<redacted>
# Default password for internal 'druid_system' user
druid.auth.authenticator.BasicMetadataAuthenticator.initialInternalClientPassword=<redacted>
# Uses the metadata store for storing users, you can use authentication API to create new users and grant permissions
druid.auth.authenticator.BasicMetadataAuthenticator.credentialsValidator.type=metadata
# If true and the request credential doesn't exists in this credentials store, the request will proceed to next Authenticator in the chain.
druid.auth.authenticator.BasicMetadataAuthenticator.skipOnFailure=true
druid.auth.authenticator.BasicMetadataAuthenticator.authorizerName=BasicMetadataAuthorizer
druid.auth.authorizers=["BasicMetadataAuthorizer"]
druid.auth.authorizer.BasicMetadataAuthorizer.type=basic
```
Let me know if you need any specific configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Communication with Coordinator is broken after upgrading to 34.0.0 (druid)
Posted by "TessaIO (via GitHub)" <gi...@apache.org>.
TessaIO commented on issue #18411:
URL: https://github.com/apache/druid/issues/18411#issuecomment-3197926473
It turns out to be the case also even in 33.0.0 version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Communication with Coordinator is broken after upgrading to 34.0.0 (druid)
Posted by "TessaIO (via GitHub)" <gi...@apache.org>.
TessaIO commented on issue #18411:
URL: https://github.com/apache/druid/issues/18411#issuecomment-3197981913
Related to https://github.com/apache/druid/issues/17961.
I think it makes sense to note this as breaking change for Kubernetes users.
Closing this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Communication with Coordinator is broken after upgrading to 34.0.0 (druid)
Posted by "TessaIO (via GitHub)" <gi...@apache.org>.
TessaIO closed issue #18411: Communication with Coordinator is broken after upgrading to 34.0.0
URL: https://github.com/apache/druid/issues/18411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org