You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Pedro Azevedo (Jira)" <ji...@apache.org> on 2024/08/22 17:46:00 UTC
[jira] [Updated] (STORM-4077) Worker being reassigned when Nimbus leadership changes
[ https://issues.apache.org/jira/browse/STORM-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pedro Azevedo updated STORM-4077:
---------------------------------
Description:
Hey guys, I'm using Storm v2.6.1 and every time I restart the nimbus leader (currently I have 3 for high availability) the workers get reassigned and this is a bad behaviour as every topology will have no workers running for a certain period(until new workers are assigned) due to a Nimbus leadership change.
On another note, when stopping the nimbus I'm getting this error which seems to be impacting the gracefully shutdown.
{code:java}
2024-08-21T15:09:46.647Z Nimbus [INFO] Shutting down master 2024-08-21T15:09:46.648Z CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting 2024-08-21T15:09:46.752Z ClientCnxn [INFO] EventThread shut down for session: 0x4000010be7caa5d 2024-08-21T15:09:46.752Z ZooKeeper [INFO] Session: 0x4000010be7caa5d closed 2024-08-21T15:09:46.752Z CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting 2024-08-21T15:09:46.812Z ProcessFunction [ERROR] Internal error processing getLeader java.lang.IllegalStateException: Expected state [STARTED] was [STOPPED] at org.apache.storm.shade.org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:835) ~[storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:462) ~[storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:507) ~[storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.zookeeper.ClientZookeeper.getChildren(ClientZookeeper.java:209) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.cluster.ZKStateStorage.get_children(ZKStateStorage.java:155) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.cluster.StormClusterStateImpl.nimbuses(StormClusterStateImpl.java:279) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.daemon.nimbus.Nimbus.getLeader(Nimbus.java:4907) ~[storm-server-2.6.1.jar:2.6.1] at org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5168) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5144) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:40) [storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:40) [storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:171) [storm-client-2.6.1.jar:2.6.1] at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492) [storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.thrift.server.Invocation.run(Invocation.java:19) [storm-shaded-deps-2.6.1.jar:2.6.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.base/java.lang.Thread.run(Thread.java:829) [?:?]{code}
was:
Hey guys, I'm using Storm v2.6.1 and every time I restart the nimbus leader (currently I have 3 for high availability) the workers get reassigned and this is a bad behaviour as every topology will have no workers running for a certain period(until new workers are assigned) due to a Nimbus leadership change.
On another note, when stopping the nimbus I'm getting this error which seems to be impacting the gracefully shutdown.
2024-08-21T15:09:46.647Z Nimbus [INFO] Shutting down master 2024-08-21T15:09:46.648Z CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting 2024-08-21T15:09:46.752Z ClientCnxn [INFO] EventThread shut down for session: 0x4000010be7caa5d 2024-08-21T15:09:46.752Z ZooKeeper [INFO] Session: 0x4000010be7caa5d closed 2024-08-21T15:09:46.752Z CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting 2024-08-21T15:09:46.812Z ProcessFunction [ERROR] Internal error processing getLeader java.lang.IllegalStateException: Expected state [STARTED] was [STOPPED] at org.apache.storm.shade.org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:835) ~[storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:462) ~[storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:507) ~[storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.zookeeper.ClientZookeeper.getChildren(ClientZookeeper.java:209) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.cluster.ZKStateStorage.get_children(ZKStateStorage.java:155) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.cluster.StormClusterStateImpl.nimbuses(StormClusterStateImpl.java:279) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.daemon.nimbus.Nimbus.getLeader(Nimbus.java:4907) ~[storm-server-2.6.1.jar:2.6.1] at org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5168) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5144) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:40) [storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:40) [storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:171) [storm-client-2.6.1.jar:2.6.1] at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492) [storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.thrift.server.Invocation.run(Invocation.java:19) [storm-shaded-deps-2.6.1.jar:2.6.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
> Worker being reassigned when Nimbus leadership changes
> ------------------------------------------------------
>
> Key: STORM-4077
> URL: https://issues.apache.org/jira/browse/STORM-4077
> Project: Apache Storm
> Issue Type: Bug
> Affects Versions: 2.6.1
> Reporter: Pedro Azevedo
> Priority: Major
>
> Hey guys, I'm using Storm v2.6.1 and every time I restart the nimbus leader (currently I have 3 for high availability) the workers get reassigned and this is a bad behaviour as every topology will have no workers running for a certain period(until new workers are assigned) due to a Nimbus leadership change.
> On another note, when stopping the nimbus I'm getting this error which seems to be impacting the gracefully shutdown.
> {code:java}
> 2024-08-21T15:09:46.647Z Nimbus [INFO] Shutting down master 2024-08-21T15:09:46.648Z CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting 2024-08-21T15:09:46.752Z ClientCnxn [INFO] EventThread shut down for session: 0x4000010be7caa5d 2024-08-21T15:09:46.752Z ZooKeeper [INFO] Session: 0x4000010be7caa5d closed 2024-08-21T15:09:46.752Z CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting 2024-08-21T15:09:46.812Z ProcessFunction [ERROR] Internal error processing getLeader java.lang.IllegalStateException: Expected state [STARTED] was [STOPPED] at org.apache.storm.shade.org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:835) ~[storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:462) ~[storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:507) ~[storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.zookeeper.ClientZookeeper.getChildren(ClientZookeeper.java:209) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.cluster.ZKStateStorage.get_children(ZKStateStorage.java:155) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.cluster.StormClusterStateImpl.nimbuses(StormClusterStateImpl.java:279) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.daemon.nimbus.Nimbus.getLeader(Nimbus.java:4907) ~[storm-server-2.6.1.jar:2.6.1] at org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5168) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5144) ~[storm-client-2.6.1.jar:2.6.1] at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:40) [storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:40) [storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:171) [storm-client-2.6.1.jar:2.6.1] at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492) [storm-shaded-deps-2.6.1.jar:2.6.1] at org.apache.storm.thrift.server.Invocation.run(Invocation.java:19) [storm-shaded-deps-2.6.1.jar:2.6.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.base/java.lang.Thread.run(Thread.java:829) [?:?]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)