You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "humutkazan (via GitHub)" <gi...@apache.org> on 2023/12/05 08:33:06 UTC
[I] Leadership Election Not Happening (druid)
humutkazan opened a new issue, #15491:
URL: https://github.com/apache/druid/issues/15491
We have encountered an issue recently and we think that it was related with leadership election not happening. You can find the coordinator log file attached.
In the logs you can see the log that says
`2023-12-01T05:06:05,659 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am the leader of the coordinators, all must bow! Starting coordination in [PT10S].`
but after some time during the day it says
`15:23,973 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am no longer the leader...`
and from this point on you could see the logs like
`2023-12-01T05:15:57,065 ERROR [KafkaSupervisor-enriched-analog-Reporting-0] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Failed to get task runner because I'm not the leader!`
connection time out exceptions etc.
[0112.log](https://github.com/apache/druid/files/13557175/0112.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Leadership Election Not Happening (druid)
Posted by "acherla (via GitHub)" <gi...@apache.org>.
acherla commented on issue #15491:
URL: https://github.com/apache/druid/issues/15491#issuecomment-1854883595
Try downgrading to 3.6.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Leadership Election Not Happening (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15491: Leadership Election Not Happening
URL: https://github.com/apache/druid/issues/15491
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Leadership Election Not Happening (druid)
Posted by "rasgele (via GitHub)" <gi...@apache.org>.
rasgele commented on issue #15491:
URL: https://github.com/apache/druid/issues/15491#issuecomment-1857555090
> Try downgrading to 3.6.x
I tried with 3.6.0 but nothing seemed to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Leadership Election Not Happening (druid)
Posted by "rasgele (via GitHub)" <gi...@apache.org>.
rasgele commented on issue #15491:
URL: https://github.com/apache/druid/issues/15491#issuecomment-1845229647
May be related to this? https://stackoverflow.com/questions/66519657/java-curator-zookeeper-client-hanging-indefinitely-when-zookeeper-is-not-availab
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Leadership Election Not Happening (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15491:
URL: https://github.com/apache/druid/issues/15491#issuecomment-2767710420
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Leadership Election Not Happening (druid)
Posted by "rasgele (via GitHub)" <gi...@apache.org>.
rasgele commented on issue #15491:
URL: https://github.com/apache/druid/issues/15491#issuecomment-1845182473
ZK is getting online then offline and then finally online. After the last successful connection, no progress was observed.
I suspect that the initial request to ZK is somehow hung, and since no timeout is set on ZK client, it waits indefinitely. Guessing this based on this logline:
`2023-12-01T05:15:53,975 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
`
Can that be the reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Leadership Election Not Happening (druid)
Posted by "rasgele (via GitHub)" <gi...@apache.org>.
rasgele commented on issue #15491:
URL: https://github.com/apache/druid/issues/15491#issuecomment-1853914644
It is 3.8.1 ("zookeeper.version" : "3.8.1-74db005175a4ec545697012f9069cb9dcc8cdda7, built on 2023-01-25 16:31 UTC")
In this case we're running a single instance of zookeeper.
Problem is reproducable by just restarting zookeeper. Even coordinator detects that zookeeper is back, it is not elected leader again.
I observer the same when I have 2 instances of coordinator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Leadership Election Not Happening (druid)
Posted by "acherla (via GitHub)" <gi...@apache.org>.
acherla commented on issue #15491:
URL: https://github.com/apache/druid/issues/15491#issuecomment-1848221932
What version of zookeeper are you using? And did you check your zookeeper quorum is actually configured correctly and stable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Leadership Election Not Happening (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15491:
URL: https://github.com/apache/druid/issues/15491#issuecomment-2840518803
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org