You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by GMTT via user <us...@storm.apache.org> on 2026/06/02 15:40:24 UTC
Upgraded Storm to version 2.8.8 and now experiencing SaslException
Hello,
We recently upgraded our Apache Storm Nimbus server to 2.8.8 from 2.8.0 and the storm-nimbus service just continuously restart with the error below. We are currently using SASL authentication using the DIGEST-MD5 mechanism between our storm and zookeeper servers which is working fine in version 2.8.0 but now we get this error in 2.8.8.
2026-06-02 17:26:29.157 o.a.s.v.ConfigValidation main [INFO] Will use [class org.apache.storm.DaemonConfig, class org.apache.storm.Config] for validation
2026-06-02 17:26:29.317 o.a.s.z.ClientZookeeper main [INFO] Starting ZK Curator
2026-06-02 17:26:29.317 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] Starting
2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:zookeeper.version=3.9.5-293c895a8d966a3ecb92872be4a1daf87d725da2, built on 2026-02-11 20:18 UTC
2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:host.name=clrv0000294762.ic.ing.net
2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.version=17.0.19
2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.vendor=Red Hat, Inc.
2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.home=/usr/lib/jvm/java-17-openjdk-17.0.19.0.10-2.el9.x86_64
2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.class.path=/var/opt/storm/apache-storm-2.8.8/*:/var/opt/storm/apache-storm-2.8.8/lib/storm-shaded-deps-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/clojure-1.12.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-util-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-handler-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-marshalling-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-exec-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-jvm-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jspecify-1.0.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/slf4j-api-2.0.17.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-core-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-annotations-2.21.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-dataformat-smile-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/hadoop-auth-3.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-lang3-3.20.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jline-2.14.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-collections4-4.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/amqp-client-5.28.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-core-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-server-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/j2objc-annotations-3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-asn1-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.activation-1.2.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/javax.annotation-api-1.3.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-io-2.22.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/httpcore-4.4.16.jar:/var/opt/storm/apache-storm-2.8.8/lib/guava-33.6.0-jre.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-pkix-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-buffer-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-logging-1.3.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-server-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/audience-annotations-0.12.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.activation-api-1.2.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/zookeeper-jute-3.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/nimbus-jose-jwt-10.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-codec-1.22.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.xml.bind-api-4.0.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-slf4j2-impl-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/zookeeper-3.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-databind-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-fileupload-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/curator-framework-5.9.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-core-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-io-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/chill-java-0.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/hadoop-shaded-guava-1.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-clojure-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-util-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-graphite-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-core-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-util-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/curator-client-5.9.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-over-slf4j-2.0.17.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-client-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/reflectasm-1.11.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-transport-native-unix-common-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/rocksdbjni-10.10.1.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/snakeyaml-2.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/spec.alpha-0.5.238.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-compression-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/error_prone_annotations-2.49.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-config-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/minlog-1.3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/kryo-5.6.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/objenesis-3.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcprov-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-security-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcpkix-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-protobuf-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-transport-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.servlet-api-6.1.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcutil-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/json-smart-2.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-session-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-core-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-common-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/asm-9.9.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/failureaccess-1.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/core.specs.alpha-0.4.74.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-http-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-base-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-api-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-resolver-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-ee10-servlet-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-ee10-servlets-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/httpclient-4.5.14.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-crypto-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/tools.logging-1.3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/accessors-smart-2.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/carbonite-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-jmx-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-compress-1.28.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-cli-1.11.0.jar:/var/opt/storm/apache-storm-2.8.8/extlib/*:/var/opt/storm/apache-storm-2.8.8/extlib-daemon/*:/var/opt/storm/current/conf
2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64
2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.io.tmpdir=/var/storm-tmp
2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.compiler=<NA>
2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.name=Linux
2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.arch=amd64
2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.version=5.14.0-611.55.1.el9_7.x86_64
2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.name=gmttusr
2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.home=/home/gmttusr
2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.dir=/
2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.memory.free=36MB
2026-06-02 17:26:29.325 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.memory.max=1024MB
2026-06-02 17:26:29.325 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.memory.total=60MB
2026-06-02 17:26:29.327 o.a.s.s.o.a.z.ZooKeeper main [INFO] Initiating client connection, connectString=clrv0000301358.ic.ing.net:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@23a9ba52
2026-06-02 17:26:29.332 o.a.s.s.o.a.z.c.X509Util main [INFO] Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2026-06-02 17:26:29.471 o.a.s.s.o.a.z.c.X509Util main [INFO] Default TLS protocol is TLSv1.3, supported TLS protocols are [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello]
2026-06-02 17:26:29.478 o.a.s.s.o.a.z.ClientCnxnSocket main [INFO] jute.maxbuffer value is 1048575 Bytes
2026-06-02 17:26:29.486 o.a.s.s.o.a.z.ClientCnxn main [INFO] zookeeper.request.timeout value is 0. feature enabled=false
2026-06-02 17:26:29.494 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] Default schema
2026-06-02 17:26:29.503 o.a.s.s.o.a.z.Login main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Client successfully logged in.
2026-06-02 17:26:29.506 o.a.s.s.o.a.z.c.ZooKeeperSaslClient main-SendThread(clrv0000301358.ic.ing.net:2181) [WARN] Client will not use DIGEST-MD5 as SASL mechanism, because FIPS mode is enabled.
2026-06-02 17:26:29.506 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Opening socket connection to server clrv0000301358.ic.ing.net/10.159.128.170:2181.
2026-06-02 17:26:29.507 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client'
2026-06-02 17:26:29.510 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Socket connection established, initiating session, client: /10.158.64.7:48370, server: clrv0000301358.ic.ing.net/10.159.128.170:2181
2026-06-02 17:26:29.515 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Session establishment complete on server clrv0000301358.ic.ing.net/10.159.128.170:2181, session id = 0x100000051861675, negotiated timeout = 20000
2026-06-02 17:26:29.516 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [ERROR] SASL authentication with Zookeeper Quorum member failed.
javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null.
at org.apache.storm.shade.org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:402)
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1212)
2026-06-02 17:26:29.518 o.a.s.s.o.a.c.f.s.ConnectionStateManager main-EventThread [INFO] State change: CONNECTED
2026-06-02 17:26:29.520 o.a.s.s.o.a.z.ClientCnxn main-EventThread [ERROR] Error while calling watcher.
java.lang.NullPointerException: Cannot invoke "org.apache.storm.shade.org.apache.zookeeper.Watcher$Event$KeeperState.ordinal()" because "state" is null
at org.apache.storm.shade.org.apache.curator.ConnectionState.checkState(ConnectionState.java:206)
at org.apache.storm.shade.org.apache.curator.ConnectionState.process(ConnectionState.java:146)
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:566)
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:541)
2026-06-02 17:26:29.521 o.a.s.s.o.a.z.ClientCnxn main-EventThread [INFO] EventThread shut down for session: 0x100000051861675
2026-06-02 17:26:29.524 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl Curator-Framework-0 [INFO] backgroundOperationsLoop exiting
2026-06-02 17:26:29.528 o.a.s.u.Utils main [ERROR] Received error in thread main.. terminating server...
java.lang.Error: org.apache.storm.shade.org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /storm
at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:668)
at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:672)
at org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:1055)
at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1082)
at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1077)
at java.base/java.lang.Thread.dispatchUncaughtException(Thread.java:2017)
Caused by: org.apache.storm.shade.org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /storm
at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:129)
at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:53)
at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1867)
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:247)
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:240)
at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:88)
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForegroundStandard(ExistsBuilderImpl.java:240)
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:235)
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:202)
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:35)
at org.apache.storm.zookeeper.AclEnforcement.verifyAcls(AclEnforcement.java:84)
at org.apache.storm.daemon.nimbus.Nimbus.launch(Nimbus.java:1608)
at org.apache.storm.daemon.nimbus.Nimbus.main(Nimbus.java:1615)
Best regards,
Norman
-----------------------------------------------------------------
ATTENTION:
The information in this e-mail is confidential and only meant for the intended recipient. If you are not the intended recipient, don't use or disclose it in any way. Please let the sender know and delete the message immediately.
-----------------------------------------------------------------
Re: Upgraded Storm to version 2.8.8 and now experiencing SaslException
Posted by Richard Zowalla <rz...@apache.org>.
Hi,
The actual issue is
"Client will not use DIGEST-MD5 as SASL mechanism, because FIPS mode is enabled."
For this reason, auth will fail later.
This was introduced in
ZooKeeper 3.9.4 introduced ZOOKEEPER-4889 - "Fallback to DIGEST-MD5 auth mech should be disabled in FIPS mode"
(https://www.mail-archive.com/issues@zookeeper.apache.org/msg09243.html). The client now refuses to create a
DIGEST-MD5 SASL client when FIPS mode is on.
Since 2.8.8. ships with a newer ZooKeeper client. The funny part is, that zookeeper.fips-mode defaults to true in ZooKeeper 3.9.x
As a workaround, you can pass the system property to the daemon JVMs in storm.yaml (it must be a -D flag; per ZOOKEEPER-4973 (https://www.mail-archive.com/issues@zookeeper.apache.org/msg09733.html) setting it via config file doesn't work for
the client side):
nimbus.childopts: "-Xmx1024m -Dzookeeper.fips-mode=false"
supervisor.childopts: "... -Dzookeeper.fips-mode=false"
# plus ui.childopts / worker.childopts if those also do SASL to ZK
The property name is not relocated by the shade plugin (it doesn't match the org.apache.zookeeper package pattern), so
plain zookeeper.fips-mode works against the shaded client.
A real fix would be migrating Storm↔ZK auth to Kerberos (GSSAPI) rather than keeping the override flag forever.
Hope it helps
Richard
> Am 02.06.2026 um 17:40 schrieb GMTT via user <us...@storm.apache.org>:
>
> Hello,
>
> We recently upgraded our Apache Storm Nimbus server to 2.8.8 from 2.8.0 and the storm-nimbus service just continuously restart with the error below. We are currently using SASL authentication using the DIGEST-MD5 mechanism between our storm and zookeeper servers which is working fine in version 2.8.0 but now we get this error in 2.8.8.
>
> 2026-06-02 17:26:29.157 o.a.s.v.ConfigValidation main [INFO] Will use [class org.apache.storm.DaemonConfig, class org.apache.storm.Config] for validation
> 2026-06-02 17:26:29.317 o.a.s.z.ClientZookeeper main [INFO] Starting ZK Curator
> 2026-06-02 17:26:29.317 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] Starting
> 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:zookeeper.version=3.9.5-293c895a8d966a3ecb92872be4a1daf87d725da2, built on 2026-02-11 20:18 UTC
> 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:host.name=clrv0000294762.ic.ing.net
> 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.version=17.0.19
> 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.vendor=Red Hat, Inc.
> 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.home=/usr/lib/jvm/java-17-openjdk-17.0.19.0.10-2.el9.x86_64
> 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.class.path=/var/opt/storm/apache-storm-2.8.8/*:/var/opt/storm/apache-storm-2.8.8/lib/storm-shaded-deps-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/clojure-1.12.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-util-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-handler-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-marshalling-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-exec-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-jvm-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jspecify-1.0.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/slf4j-api-2.0.17.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-core-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-annotations-2.21.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-dataformat-smile-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/hadoop-auth-3.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-lang3-3.20.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jline-2.14.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-collections4-4.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/amqp-client-5.28.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-core-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-server-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/j2objc-annotations-3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-asn1-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.activation-1.2.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/javax.annotation-api-1.3.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-io-2.22.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/httpcore-4.4.16.jar:/var/opt/storm/apache-storm-2.8.8/lib/guava-33.6.0-jre.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-pkix-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-buffer-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-logging-1.3.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-server-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/audience-annotations-0.12.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.activation-api-1.2.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/zookeeper-jute-3.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/nimbus-jose-jwt-10.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-codec-1.22.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.xml.bind-api-4.0.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-slf4j2-impl-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/zookeeper-3.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-databind-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-fileupload-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/curator-framework-5.9.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-core-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-io-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/chill-java-0.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/hadoop-shaded-guava-1.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-clojure-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-util-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-graphite-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-core-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-util-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/curator-client-5.9.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-over-slf4j-2.0.17.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-client-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/reflectasm-1.11.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-transport-native-unix-common-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/rocksdbjni-10.10.1.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/snakeyaml-2.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/spec.alpha-0.5.238.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-compression-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/error_prone_annotations-2.49.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-config-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/minlog-1.3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/kryo-5.6.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/objenesis-3.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcprov-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-security-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcpkix-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-protobuf-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-transport-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.servlet-api-6.1.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcutil-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/json-smart-2.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-session-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-core-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-common-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/asm-9.9.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/failureaccess-1.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/core.specs.alpha-0.4.74.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-http-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-base-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-api-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-resolver-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-ee10-servlet-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-ee10-servlets-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/httpclient-4.5.14.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-crypto-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/tools.logging-1.3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/accessors-smart-2.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/carbonite-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-jmx-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-compress-1.28.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-cli-1.11.0.jar:/var/opt/storm/apache-storm-2.8.8/extlib/*:/var/opt/storm/apache-storm-2.8.8/extlib-daemon/*:/var/opt/storm/current/conf
> 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64
> 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.io.tmpdir=/var/storm-tmp
> 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.compiler=<NA>
> 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.name=Linux
> 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.arch=amd64
> 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.version=5.14.0-611.55.1.el9_7.x86_64
> 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.name=gmttusr
> 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.home=/home/gmttusr
> 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.dir=/
> 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.memory.free=36MB
> 2026-06-02 17:26:29.325 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.memory.max=1024MB
> 2026-06-02 17:26:29.325 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.memory.total=60MB
> 2026-06-02 17:26:29.327 o.a.s.s.o.a.z.ZooKeeper main [INFO] Initiating client connection, connectString=clrv0000301358.ic.ing.net:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@23a9ba52
> 2026-06-02 17:26:29.332 o.a.s.s.o.a.z.c.X509Util main [INFO] Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
> 2026-06-02 17:26:29.471 o.a.s.s.o.a.z.c.X509Util main [INFO] Default TLS protocol is TLSv1.3, supported TLS protocols are [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello]
> 2026-06-02 17:26:29.478 o.a.s.s.o.a.z.ClientCnxnSocket main [INFO] jute.maxbuffer value is 1048575 Bytes
> 2026-06-02 17:26:29.486 o.a.s.s.o.a.z.ClientCnxn main [INFO] zookeeper.request.timeout value is 0. feature enabled=false
> 2026-06-02 17:26:29.494 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] Default schema
> 2026-06-02 17:26:29.503 o.a.s.s.o.a.z.Login main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Client successfully logged in.
> 2026-06-02 17:26:29.506 o.a.s.s.o.a.z.c.ZooKeeperSaslClient main-SendThread(clrv0000301358.ic.ing.net:2181) [WARN] Client will not use DIGEST-MD5 as SASL mechanism, because FIPS mode is enabled.
> 2026-06-02 17:26:29.506 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Opening socket connection to server clrv0000301358.ic.ing.net/10.159.128.170:2181.
> 2026-06-02 17:26:29.507 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client'
> 2026-06-02 17:26:29.510 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Socket connection established, initiating session, client: /10.158.64.7:48370, server: clrv0000301358.ic.ing.net/10.159.128.170:2181
> 2026-06-02 17:26:29.515 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Session establishment complete on server clrv0000301358.ic.ing.net/10.159.128.170:2181, session id = 0x100000051861675, negotiated timeout = 20000
> 2026-06-02 17:26:29.516 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [ERROR] SASL authentication with Zookeeper Quorum member failed.
> javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null.
> at org.apache.storm.shade.org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:402)
> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1212)
> 2026-06-02 17:26:29.518 o.a.s.s.o.a.c.f.s.ConnectionStateManager main-EventThread [INFO] State change: CONNECTED
> 2026-06-02 17:26:29.520 o.a.s.s.o.a.z.ClientCnxn main-EventThread [ERROR] Error while calling watcher.
> java.lang.NullPointerException: Cannot invoke "org.apache.storm.shade.org.apache.zookeeper.Watcher$Event$KeeperState.ordinal()" because "state" is null
> at org.apache.storm.shade.org.apache.curator.ConnectionState.checkState(ConnectionState.java:206)
> at org.apache.storm.shade.org.apache.curator.ConnectionState.process(ConnectionState.java:146)
> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:566)
> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:541)
> 2026-06-02 17:26:29.521 o.a.s.s.o.a.z.ClientCnxn main-EventThread [INFO] EventThread shut down for session: 0x100000051861675
> 2026-06-02 17:26:29.524 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl Curator-Framework-0 [INFO] backgroundOperationsLoop exiting
> 2026-06-02 17:26:29.528 o.a.s.u.Utils main [ERROR] Received error in thread main.. terminating server...
> java.lang.Error: org.apache.storm.shade.org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /storm
> at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:668)
> at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:672)
> at org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:1055)
> at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1082)
> at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1077)
> at java.base/java.lang.Thread.dispatchUncaughtException(Thread.java:2017)
> Caused by: org.apache.storm.shade.org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /storm
> at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:129)
> at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:53)
> at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1867)
> at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:247)
> at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:240)
> at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:88)
> at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForegroundStandard(ExistsBuilderImpl.java:240)
> at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:235)
> at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:202)
> at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:35)
> at org.apache.storm.zookeeper.AclEnforcement.verifyAcls(AclEnforcement.java:84)
> at org.apache.storm.daemon.nimbus.Nimbus.launch(Nimbus.java:1608)
> at org.apache.storm.daemon.nimbus.Nimbus.main(Nimbus.java:1615)
>
> Best regards,
> Norman
> -----------------------------------------------------------------
> ATTENTION:
> The information in this e-mail is confidential and only meant for the intended recipient. If you are not the intended recipient, don't use or disclose it in any way. Please let the sender know and delete the message immediately.
> -----------------------------------------------------------------