You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Shields, Paul" <pa...@hpe.com.INVALID> on 2025/10/24 20:18:54 UTC
MQTT Last Will not sent because denied authentication
Hi,
Why are MQTT last will messages authenticated with Artemis before being sent? From my understanding of the MQTT last will feature is that authentication is a separate step handled during the client's initial connection. The LWT itself is a message prepared by a client and stored by the broker to be published only if the client disconnects unexpectedly. The security of the LWT message is therefore dependent on the security of the initial client connection, which must be established through methods like username/password or TLS and not when the LWT is delivered/published.
We have written a custom securityManager plugin that uses Json Web Tokens as passwords for connecting MQTT clients. We use MQTT for server availability which has a state of either up or down. When a server client connects with the MQTT Broker it is authenticated with the broker using a JWT, registers a last will, and then publishes a server up message on a MQTT topic. We have other MQTT clients that also connect with the Artemis broker using JWTs for auth and then subscribe to server state topics. The MQTT last will is used to publish server down messages when the long running server dies for some reason. Some of the servers publish “re-announce” messages as they complete certain steps of processing that they publish on a different topic using the initial client connection. This was initially developed using Artemis 2.28.0 and later used in Artemis 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager, implements the ActiveMQSecurityManager5 interface. During our initial development of the securityManager plugin we saw MQTT last will messages failing authentication and put a workaround in place to check that if there is a subject and a principal with the same user name (each MQTT client has a unique user name) which means that this is the same session that has previously been authenticated, we will validate the JWT, but ignore the expiration time. We are now upgrading to Artemis 2.43.0 and would like to remove the workaround. Our workaround for last will authentication breaks down when the keys are rotated. It is about to become a real problem as we are going to rotate the JWT key on a more frequent schedule. Here are the errors produced:
2025-10-14 15:06:51,858 INFO [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed: username: x3000c0s9b0n0, details: auth is invalid
InvalidJwtException: JWT processing failed. Additional details: [[17] Unable to process JOSE object (cause: org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable verification key for JWS w/ header {"alg":"RS256","kid”:”previous KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED, e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C, alg=RS256, n=CENSORED, e=CENSORED}]): JsonWebSignature{"alg":"RS256","kid”:”previous KEY-A","typ":”JWT”CENSORED
]
2025-10-14 15:06:51,861 INFO [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication for user: x3000c0s9b0n0, subject: Subject:
Principal: pzxQ2XQN
Principal: admin
Principal: x3000c0s9b0n0
2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server] AMQ222216: Security problem while authenticating: AMQ229031: Unable to validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN: unavailable
2025-10-14 15:06:51,863 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH, isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67], variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state, packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap: 27/27, unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
org.apache.activemq.artemis.api.core.ActiveMQSecurityException: AMQ229031: Unable to validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN: unavailable
at org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
at org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server] AMQ222216: Security problem while authenticating: AMQ229031: Unable to validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN: unavailable
2025-10-14 15:06:51,866 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007: Authorization failure sending will message: AMQ229031: Unable to validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN: unavailable
Regards,
Paul Shields
Re: MQTT Last Will not sent because denied authentication
Posted by Matt Pavlovich <ma...@apache.org>.
This approach has always been a challenge for token-based authn systems, it isn’t JWT specific. A long running stateful connection is used for performance and lowering latency (usually queues, databases, etc).
Feels like back-end systems are an afterthought for JWT system design, or rather JWT presumes REST/API style connection-per-request behavior.
Two-way SSL certificates tend to split the difference.
-Matt
> On Oct 28, 2025, at 2:12 PM, Shields, Paul <pa...@hpe.com.INVALID> wrote:
>
> Hi Matt,
>
> Thank you for the suggestion, But our backend s/w design is based around long term connections, So open and closing the connection has consequences in our backend. The JWT works well for the authenticate step of gaining access to the broker and setting up a session. It’s our authorize method that needs work. Does a session live as long as the MQTT client is connected to the broker? Any docs on how sessions are used and how they interact with the authorize method would be helpful.
>
> Regards,
> Paul
>
> From: Matt Pavlovich <ma...@apache.org>
> Date: Tuesday, October 28, 2025 at 1:59 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: MQTT Last Will not sent because denied authentication
>
> Hi Paul-
>
> The usage of JWT and LWT are competing features, since JWT expires and LWT is intended to alert for unplanned disconnect of long-running connections. A possible solution is to perform periodic close-open of the connections and re-register the LWT send would always fall within the timeframe of the JWT expiry.
>
> Matt Pavlovich
>
>> On Oct 28, 2025, at 11:36 AM, Shields, Paul <pa...@hpe.com.INVALID> wrote:
>>
>> Hi Justin,
>>
>> I am struggling with this statement
>> However, authorization for actually
>> sending the will message is not performed at this point. Authorization is
>> only performed when the LWT message is actually sent (e.g. when the
>> client's connection fails). All sending operations are authorized at the
>> time they occur whether that's via the normal PUBLISH packet from the
>> client or for a LWT message sent on the client's behalf by the broker.
>> If we were using a user name and password (even if that was stored in LDAP) that would be ok. But we are using the machine ID for the user name and a JWT is used in the password field of the MQTT connect. My problem is that the JWTs for authentication have a short expiration time (5min). This link explains the JWT concept we are using for authentication. https://urldefense.com/v3/__https://stytch.com/blog/understanding-jwks/__;!!NpxR!l0K48Qr3zBCaoFmyCZXVvGQOZMZY59F9n8cgHM4q5vNc5xz0PCpg7C4HOGannJ2cB0tn2U7ian9s7tNc7A$ but instead of using Stytch as the authorization server JWKS https endpoint we are using Spire with a plugin for the JWKS. The specific issue is that the JWT supplied with the connect to the broker has expired by the time the LWT is being delivered. We chose the MQTT protocol for its simplicity and the LWT feature for server heartbeat.
>>
>> Not sure we have implemented the authorize function of the securityManager class in alignment with how Artemis operates. We do not create users up front but on the fly as each machine connects to publish its status or scribe to another's status. We use the admin role for all of the machine users/IDs. One on of the key features for using the JWT for us is that we imbed the machine ID in the jwt payload and in the authorize function reject any publish (SEND) request if the target topic does not match the sender machine ID. Not sure how we would update the JWT for every connection every 5min so that the JWT would be valid at the time it will be sent?
>>
>> Is there a reference implementation of a pluggable securityManager that uses JWTs?
>>
>> Any other suggestions?
>>
>> Regards,
>> Paul
>>
>> From: Justin Bertram <jb...@apache.org>
>> Date: Tuesday, October 28, 2025 at 9:34 AM
>> To: users@activemq.apache.org <us...@activemq.apache.org>
>> Subject: Re: MQTT Last Will not sent because denied authentication
>>
>> Just following up to ensure my explanation made sense. Do you need anything
>> further here?
>>
>>
>> Justin
>>
>> On Fri, Oct 24, 2025 at 4:17 PM Justin Bertram <jb...@apache.org> wrote:
>>
>>> An MQTT client connects to the broker via a CONNECT packet. Typically this
>>> packet contains the client's credentials which are then authenticated by
>>> the broker.
>>>
>>> Keep in mind that authentication and authorization are related but
>>> separate things. A client may be authenticated but not authorized to
>>> consume messages from or send messages to specific topics. This is a
>>> fundamental concept of role-based access control.
>>>
>>> The CONNECT packet also contains all the details about the LWT message
>>> (i.e. payload, properties, & topic). However, authorization for actually
>>> sending the will message is not performed at this point. Authorization is
>>> only performed when the LWT message is actually sent (e.g. when the
>>> client's connection fails). All sending operations are authorized at the
>>> time they occur whether that's via the normal PUBLISH packet from the
>>> client or for a LWT message sent on the client's behalf by the broker.
>>>
>>> Authorization is important here because the destination topic for the LWT
>>> message is arbitrary. If authorization was not performed then it would be
>>> simple for a client to send a message to a topic which it would not
>>> otherwise have authorization.
>>>
>>> If authorization is failing when the broker attempts to send the client's
>>> LWT message and you're circumventing this so that the message is actually
>>> sent then it seems you may be undermining the security of your environment.
>>> You may inadvertently allow a clever, nefarious actor to send a message to
>>> a topic to which they are not authorized.
>>>
>>> How are you currently re-authenticating your clients when their JWTs
>>> expire?
>>>
>>>
>>> Justin
>>>
>>> On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul <pa...@hpe.com.invalid>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Why are MQTT last will messages authenticated with Artemis before being
>>>> sent? From my understanding of the MQTT last will feature is that
>>>> authentication is a separate step handled during the client's initial
>>>> connection. The LWT itself is a message prepared by a client and stored by
>>>> the broker to be published only if the client disconnects unexpectedly. The
>>>> security of the LWT message is therefore dependent on the security of the
>>>> initial client connection, which must be established through methods like
>>>> username/password or TLS and not when the LWT is delivered/published.
>>>>
>>>> We have written a custom securityManager plugin that uses Json Web
>>>> Tokens as passwords for connecting MQTT clients. We use MQTT for server
>>>> availability which has a state of either up or down. When a server client
>>>> connects with the MQTT Broker it is authenticated with the broker using a
>>>> JWT, registers a last will, and then publishes a server up message on a
>>>> MQTT topic. We have other MQTT clients that also connect with the Artemis
>>>> broker using JWTs for auth and then subscribe to server state topics. The
>>>> MQTT last will is used to publish server down messages when the long
>>>> running server dies for some reason. Some of the servers publish
>>>> “re-announce” messages as they complete certain steps of processing that
>>>> they publish on a different topic using the initial client connection.
>>>> This was initially developed using Artemis 2.28.0 and later used in Artemis
>>>> 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager,
>>>> implements the ActiveMQSecurityManager5 interface. During our initial
>>>> development of the securityManager plugin we saw MQTT last will messages
>>>> failing authentication and put a workaround in place to check that if there
>>>> is a subject and a principal with the same user name (each MQTT client has
>>>> a unique user name) which means that this is the same session that has
>>>> previously been authenticated, we will validate the JWT, but ignore the
>>>> expiration time. We are now upgrading to Artemis 2.43.0 and would like to
>>>> remove the workaround. Our workaround for last will authentication breaks
>>>> down when the keys are rotated. It is about to become a real problem as we
>>>> are going to rotate the JWT key on a more frequent schedule. Here are the
>>>> errors produced:
>>>>
>>>> 2025-10-14 15:06:51,858 INFO
>>>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed: username:
>>>> x3000c0s9b0n0, details: auth is invalid
>>>> InvalidJwtException: JWT processing failed. Additional details: [[17]
>>>> Unable to process JOSE object (cause:
>>>> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
>>>> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
>>>> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
>>>> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
>>>> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED,
>>>> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C, alg=RS256,
>>>> n=CENSORED, e=CENSORED}]): JsonWebSignature{"alg":"RS256","kid”:”previous
>>>> KEY-A","typ":”JWT”CENSORED
>>>> ]
>>>> 2025-10-14 15:06:51,861 INFO
>>>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication for
>>>> user: x3000c0s9b0n0, subject: Subject:
>>>> Principal: pzxQ2XQN
>>>> Principal: admin
>>>> Principal: x3000c0s9b0n0
>>>>
>>>> 2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server]
>>>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>>>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>>>> certificate subject DN: unavailable
>>>> 2025-10-14 15:06:51,863 ERROR
>>>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
>>>> processing control packet:
>>>> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
>>>> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67],
>>>> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
>>>> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap: 27/27,
>>>> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
>>>> org.apache.activemq.artemis.api.core.ActiveMQSecurityException:
>>>> AMQ229031: Unable to validate user from 127.0.0.6:49859. Username:
>>>> x3000c0s9b0n0; SSL certificate subject DN: unavailable
>>>> at
>>>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
>>>> at
>>>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
>>>> at
>>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
>>>> at
>>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
>>>> at
>>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
>>>> at
>>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
>>>> at
>>>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
>>>> at
>>>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
>>>> at
>>>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
>>>> at
>>>> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
>>>> at
>>>> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>>>> at
>>>> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
>>>> 2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server]
>>>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>>>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>>>> certificate subject DN: unavailable
>>>> 2025-10-14 15:06:51,866 ERROR
>>>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007: Authorization
>>>> failure sending will message: AMQ229031: Unable to validate user from
>>>> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN:
>>>> unavailable
>>>>
>>>> Regards,
>>>> Paul Shields
>>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://urldefense.com/v3/__https://activemq.apache.org/contact__;!!NpxR!l0K48Qr3zBCaoFmyCZXVvGQOZMZY59F9n8cgHM4q5vNc5xz0PCpg7C4HOGannJ2cB0tn2U7ian-lWM-jFw$
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: MQTT Last Will not sent because denied authentication
Posted by "Shields, Paul" <pa...@hpe.com.INVALID>.
Hi Matt,
Thank you for the suggestion, But our backend s/w design is based around long term connections, So open and closing the connection has consequences in our backend. The JWT works well for the authenticate step of gaining access to the broker and setting up a session. It’s our authorize method that needs work. Does a session live as long as the MQTT client is connected to the broker? Any docs on how sessions are used and how they interact with the authorize method would be helpful.
Regards,
Paul
From: Matt Pavlovich <ma...@apache.org>
Date: Tuesday, October 28, 2025 at 1:59 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: MQTT Last Will not sent because denied authentication
Hi Paul-
The usage of JWT and LWT are competing features, since JWT expires and LWT is intended to alert for unplanned disconnect of long-running connections. A possible solution is to perform periodic close-open of the connections and re-register the LWT send would always fall within the timeframe of the JWT expiry.
Matt Pavlovich
> On Oct 28, 2025, at 11:36 AM, Shields, Paul <pa...@hpe.com.INVALID> wrote:
>
> Hi Justin,
>
> I am struggling with this statement
> However, authorization for actually
> sending the will message is not performed at this point. Authorization is
> only performed when the LWT message is actually sent (e.g. when the
> client's connection fails). All sending operations are authorized at the
> time they occur whether that's via the normal PUBLISH packet from the
> client or for a LWT message sent on the client's behalf by the broker.
> If we were using a user name and password (even if that was stored in LDAP) that would be ok. But we are using the machine ID for the user name and a JWT is used in the password field of the MQTT connect. My problem is that the JWTs for authentication have a short expiration time (5min). This link explains the JWT concept we are using for authentication. https://urldefense.com/v3/__https://stytch.com/blog/understanding-jwks/__;!!NpxR!l0K48Qr3zBCaoFmyCZXVvGQOZMZY59F9n8cgHM4q5vNc5xz0PCpg7C4HOGannJ2cB0tn2U7ian9s7tNc7A$ but instead of using Stytch as the authorization server JWKS https endpoint we are using Spire with a plugin for the JWKS. The specific issue is that the JWT supplied with the connect to the broker has expired by the time the LWT is being delivered. We chose the MQTT protocol for its simplicity and the LWT feature for server heartbeat.
>
> Not sure we have implemented the authorize function of the securityManager class in alignment with how Artemis operates. We do not create users up front but on the fly as each machine connects to publish its status or scribe to another's status. We use the admin role for all of the machine users/IDs. One on of the key features for using the JWT for us is that we imbed the machine ID in the jwt payload and in the authorize function reject any publish (SEND) request if the target topic does not match the sender machine ID. Not sure how we would update the JWT for every connection every 5min so that the JWT would be valid at the time it will be sent?
>
> Is there a reference implementation of a pluggable securityManager that uses JWTs?
>
> Any other suggestions?
>
> Regards,
> Paul
>
> From: Justin Bertram <jb...@apache.org>
> Date: Tuesday, October 28, 2025 at 9:34 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: MQTT Last Will not sent because denied authentication
>
> Just following up to ensure my explanation made sense. Do you need anything
> further here?
>
>
> Justin
>
> On Fri, Oct 24, 2025 at 4:17 PM Justin Bertram <jb...@apache.org> wrote:
>
>> An MQTT client connects to the broker via a CONNECT packet. Typically this
>> packet contains the client's credentials which are then authenticated by
>> the broker.
>>
>> Keep in mind that authentication and authorization are related but
>> separate things. A client may be authenticated but not authorized to
>> consume messages from or send messages to specific topics. This is a
>> fundamental concept of role-based access control.
>>
>> The CONNECT packet also contains all the details about the LWT message
>> (i.e. payload, properties, & topic). However, authorization for actually
>> sending the will message is not performed at this point. Authorization is
>> only performed when the LWT message is actually sent (e.g. when the
>> client's connection fails). All sending operations are authorized at the
>> time they occur whether that's via the normal PUBLISH packet from the
>> client or for a LWT message sent on the client's behalf by the broker.
>>
>> Authorization is important here because the destination topic for the LWT
>> message is arbitrary. If authorization was not performed then it would be
>> simple for a client to send a message to a topic which it would not
>> otherwise have authorization.
>>
>> If authorization is failing when the broker attempts to send the client's
>> LWT message and you're circumventing this so that the message is actually
>> sent then it seems you may be undermining the security of your environment.
>> You may inadvertently allow a clever, nefarious actor to send a message to
>> a topic to which they are not authorized.
>>
>> How are you currently re-authenticating your clients when their JWTs
>> expire?
>>
>>
>> Justin
>>
>> On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul <pa...@hpe.com.invalid>
>> wrote:
>>
>>> Hi,
>>>
>>> Why are MQTT last will messages authenticated with Artemis before being
>>> sent? From my understanding of the MQTT last will feature is that
>>> authentication is a separate step handled during the client's initial
>>> connection. The LWT itself is a message prepared by a client and stored by
>>> the broker to be published only if the client disconnects unexpectedly. The
>>> security of the LWT message is therefore dependent on the security of the
>>> initial client connection, which must be established through methods like
>>> username/password or TLS and not when the LWT is delivered/published.
>>>
>>> We have written a custom securityManager plugin that uses Json Web
>>> Tokens as passwords for connecting MQTT clients. We use MQTT for server
>>> availability which has a state of either up or down. When a server client
>>> connects with the MQTT Broker it is authenticated with the broker using a
>>> JWT, registers a last will, and then publishes a server up message on a
>>> MQTT topic. We have other MQTT clients that also connect with the Artemis
>>> broker using JWTs for auth and then subscribe to server state topics. The
>>> MQTT last will is used to publish server down messages when the long
>>> running server dies for some reason. Some of the servers publish
>>> “re-announce” messages as they complete certain steps of processing that
>>> they publish on a different topic using the initial client connection.
>>> This was initially developed using Artemis 2.28.0 and later used in Artemis
>>> 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager,
>>> implements the ActiveMQSecurityManager5 interface. During our initial
>>> development of the securityManager plugin we saw MQTT last will messages
>>> failing authentication and put a workaround in place to check that if there
>>> is a subject and a principal with the same user name (each MQTT client has
>>> a unique user name) which means that this is the same session that has
>>> previously been authenticated, we will validate the JWT, but ignore the
>>> expiration time. We are now upgrading to Artemis 2.43.0 and would like to
>>> remove the workaround. Our workaround for last will authentication breaks
>>> down when the keys are rotated. It is about to become a real problem as we
>>> are going to rotate the JWT key on a more frequent schedule. Here are the
>>> errors produced:
>>>
>>> 2025-10-14 15:06:51,858 INFO
>>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed: username:
>>> x3000c0s9b0n0, details: auth is invalid
>>> InvalidJwtException: JWT processing failed. Additional details: [[17]
>>> Unable to process JOSE object (cause:
>>> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
>>> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
>>> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
>>> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
>>> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED,
>>> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C, alg=RS256,
>>> n=CENSORED, e=CENSORED}]): JsonWebSignature{"alg":"RS256","kid”:”previous
>>> KEY-A","typ":”JWT”CENSORED
>>> ]
>>> 2025-10-14 15:06:51,861 INFO
>>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication for
>>> user: x3000c0s9b0n0, subject: Subject:
>>> Principal: pzxQ2XQN
>>> Principal: admin
>>> Principal: x3000c0s9b0n0
>>>
>>> 2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server]
>>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>>> certificate subject DN: unavailable
>>> 2025-10-14 15:06:51,863 ERROR
>>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
>>> processing control packet:
>>> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
>>> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67],
>>> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
>>> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap: 27/27,
>>> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
>>> org.apache.activemq.artemis.api.core.ActiveMQSecurityException:
>>> AMQ229031: Unable to validate user from 127.0.0.6:49859. Username:
>>> x3000c0s9b0n0; SSL certificate subject DN: unavailable
>>> at
>>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
>>> at
>>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
>>> at
>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
>>> at
>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
>>> at
>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
>>> at
>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
>>> at
>>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
>>> at
>>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
>>> at
>>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
>>> at
>>> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
>>> at
>>> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>>> at
>>> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
>>> 2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server]
>>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>>> certificate subject DN: unavailable
>>> 2025-10-14 15:06:51,866 ERROR
>>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007: Authorization
>>> failure sending will message: AMQ229031: Unable to validate user from
>>> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN:
>>> unavailable
>>>
>>> Regards,
>>> Paul Shields
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://urldefense.com/v3/__https://activemq.apache.org/contact__;!!NpxR!l0K48Qr3zBCaoFmyCZXVvGQOZMZY59F9n8cgHM4q5vNc5xz0PCpg7C4HOGannJ2cB0tn2U7ian-lWM-jFw$
Re: MQTT Last Will not sent because denied authentication
Posted by Matt Pavlovich <ma...@apache.org>.
Yeah, iirc ActiveMQ Classic uses the same session and security context from initial connect for the LWT delivery in MQTT.
> On Oct 28, 2025, at 2:19 PM, Justin Bertram <jb...@apache.org> wrote:
>
> I tend to agree with Matt here, but I think perhaps a feature could be
> added to Artemis so that LWT are authorized on connect to avoid this kind
> of problem. This behavior would be off by default so as not to impact
> existing users.
>
> Do you have a Jira account? If so, would you mind opening an issue [1] for
> this?
>
>
> Justin
>
> [1] https://issues.apache.org/jira/browse/ARTEMIS
>
> On Tue, Oct 28, 2025 at 1:58 PM Matt Pavlovich <ma...@apache.org> wrote:
>
>> Hi Paul-
>>
>> The usage of JWT and LWT are competing features, since JWT expires and LWT
>> is intended to alert for unplanned disconnect of long-running connections.
>> A possible solution is to perform periodic close-open of the connections
>> and re-register the LWT send would always fall within the timeframe of the
>> JWT expiry.
>>
>> Matt Pavlovich
>>
>>> On Oct 28, 2025, at 11:36 AM, Shields, Paul <pa...@hpe.com.INVALID>
>> wrote:
>>>
>>> Hi Justin,
>>>
>>> I am struggling with this statement
>>> However, authorization for actually
>>> sending the will message is not performed at this point. Authorization is
>>> only performed when the LWT message is actually sent (e.g. when the
>>> client's connection fails). All sending operations are authorized at the
>>> time they occur whether that's via the normal PUBLISH packet from the
>>> client or for a LWT message sent on the client's behalf by the broker.
>>> If we were using a user name and password (even if that was stored in
>> LDAP) that would be ok. But we are using the machine ID for the user name
>> and a JWT is used in the password field of the MQTT connect. My problem is
>> that the JWTs for authentication have a short expiration time (5min). This
>> link explains the JWT concept we are using for authentication.
>> https://stytch.com/blog/understanding-jwks/ but instead of using Stytch
>> as the authorization server JWKS https endpoint we are using Spire with a
>> plugin for the JWKS. The specific issue is that the JWT supplied with the
>> connect to the broker has expired by the time the LWT is being delivered.
>> We chose the MQTT protocol for its simplicity and the LWT feature for
>> server heartbeat.
>>>
>>> Not sure we have implemented the authorize function of the
>> securityManager class in alignment with how Artemis operates. We do not
>> create users up front but on the fly as each machine connects to publish
>> its status or scribe to another's status. We use the admin role for all of
>> the machine users/IDs. One on of the key features for using the JWT for us
>> is that we imbed the machine ID in the jwt payload and in the authorize
>> function reject any publish (SEND) request if the target topic does not
>> match the sender machine ID. Not sure how we would update the JWT for every
>> connection every 5min so that the JWT would be valid at the time it will be
>> sent?
>>>
>>> Is there a reference implementation of a pluggable securityManager that
>> uses JWTs?
>>>
>>> Any other suggestions?
>>>
>>> Regards,
>>> Paul
>>>
>>> From: Justin Bertram <jb...@apache.org>
>>> Date: Tuesday, October 28, 2025 at 9:34 AM
>>> To: users@activemq.apache.org <us...@activemq.apache.org>
>>> Subject: Re: MQTT Last Will not sent because denied authentication
>>>
>>> Just following up to ensure my explanation made sense. Do you need
>> anything
>>> further here?
>>>
>>>
>>> Justin
>>>
>>> On Fri, Oct 24, 2025 at 4:17 PM Justin Bertram <jb...@apache.org>
>> wrote:
>>>
>>>> An MQTT client connects to the broker via a CONNECT packet. Typically
>> this
>>>> packet contains the client's credentials which are then authenticated by
>>>> the broker.
>>>>
>>>> Keep in mind that authentication and authorization are related but
>>>> separate things. A client may be authenticated but not authorized to
>>>> consume messages from or send messages to specific topics. This is a
>>>> fundamental concept of role-based access control.
>>>>
>>>> The CONNECT packet also contains all the details about the LWT message
>>>> (i.e. payload, properties, & topic). However, authorization for actually
>>>> sending the will message is not performed at this point. Authorization
>> is
>>>> only performed when the LWT message is actually sent (e.g. when the
>>>> client's connection fails). All sending operations are authorized at the
>>>> time they occur whether that's via the normal PUBLISH packet from the
>>>> client or for a LWT message sent on the client's behalf by the broker.
>>>>
>>>> Authorization is important here because the destination topic for the
>> LWT
>>>> message is arbitrary. If authorization was not performed then it would
>> be
>>>> simple for a client to send a message to a topic which it would not
>>>> otherwise have authorization.
>>>>
>>>> If authorization is failing when the broker attempts to send the
>> client's
>>>> LWT message and you're circumventing this so that the message is
>> actually
>>>> sent then it seems you may be undermining the security of your
>> environment.
>>>> You may inadvertently allow a clever, nefarious actor to send a message
>> to
>>>> a topic to which they are not authorized.
>>>>
>>>> How are you currently re-authenticating your clients when their JWTs
>>>> expire?
>>>>
>>>>
>>>> Justin
>>>>
>>>> On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul
>> <pa...@hpe.com.invalid>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Why are MQTT last will messages authenticated with Artemis before being
>>>>> sent? From my understanding of the MQTT last will feature is that
>>>>> authentication is a separate step handled during the client's initial
>>>>> connection. The LWT itself is a message prepared by a client and
>> stored by
>>>>> the broker to be published only if the client disconnects
>> unexpectedly. The
>>>>> security of the LWT message is therefore dependent on the security of
>> the
>>>>> initial client connection, which must be established through methods
>> like
>>>>> username/password or TLS and not when the LWT is delivered/published.
>>>>>
>>>>> We have written a custom securityManager plugin that uses Json Web
>>>>> Tokens as passwords for connecting MQTT clients. We use MQTT for server
>>>>> availability which has a state of either up or down. When a server
>> client
>>>>> connects with the MQTT Broker it is authenticated with the broker
>> using a
>>>>> JWT, registers a last will, and then publishes a server up message on a
>>>>> MQTT topic. We have other MQTT clients that also connect with the
>> Artemis
>>>>> broker using JWTs for auth and then subscribe to server state topics.
>> The
>>>>> MQTT last will is used to publish server down messages when the long
>>>>> running server dies for some reason. Some of the servers publish
>>>>> “re-announce” messages as they complete certain steps of processing
>> that
>>>>> they publish on a different topic using the initial client connection.
>>>>> This was initially developed using Artemis 2.28.0 and later used in
>> Artemis
>>>>> 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager,
>>>>> implements the ActiveMQSecurityManager5 interface. During our initial
>>>>> development of the securityManager plugin we saw MQTT last will
>> messages
>>>>> failing authentication and put a workaround in place to check that if
>> there
>>>>> is a subject and a principal with the same user name (each MQTT client
>> has
>>>>> a unique user name) which means that this is the same session that has
>>>>> previously been authenticated, we will validate the JWT, but ignore the
>>>>> expiration time. We are now upgrading to Artemis 2.43.0 and would
>> like to
>>>>> remove the workaround. Our workaround for last will authentication
>> breaks
>>>>> down when the keys are rotated. It is about to become a real problem
>> as we
>>>>> are going to rotate the JWT key on a more frequent schedule. Here are
>> the
>>>>> errors produced:
>>>>>
>>>>> 2025-10-14 15:06:51,858 INFO
>>>>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed:
>> username:
>>>>> x3000c0s9b0n0, details: auth is invalid
>>>>> InvalidJwtException: JWT processing failed. Additional details: [[17]
>>>>> Unable to process JOSE object (cause:
>>>>> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
>>>>> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
>>>>> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
>>>>> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
>>>>> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED,
>>>>> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C,
>> alg=RS256,
>>>>> n=CENSORED, e=CENSORED}]):
>> JsonWebSignature{"alg":"RS256","kid”:”previous
>>>>> KEY-A","typ":”JWT”CENSORED
>>>>> ]
>>>>> 2025-10-14 15:06:51,861 INFO
>>>>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication
>> for
>>>>> user: x3000c0s9b0n0, subject: Subject:
>>>>> Principal: pzxQ2XQN
>>>>> Principal: admin
>>>>> Principal: x3000c0s9b0n0
>>>>>
>>>>> 2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server]
>>>>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>>>>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>>>>> certificate subject DN: unavailable
>>>>> 2025-10-14 15:06:51,863 ERROR
>>>>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
>>>>> processing control packet:
>>>>> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
>>>>> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67],
>>>>>
>> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
>>>>> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap:
>> 27/27,
>>>>> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
>>>>> org.apache.activemq.artemis.api.core.ActiveMQSecurityException:
>>>>> AMQ229031: Unable to validate user from 127.0.0.6:49859. Username:
>>>>> x3000c0s9b0n0; SSL certificate subject DN: unavailable
>>>>> at
>>>>>
>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
>>>>> at
>>>>> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
>>>>> at
>>>>>
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>>>>> at
>>>>>
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>>>>> at
>>>>>
>> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
>>>>> 2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server]
>>>>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>>>>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>>>>> certificate subject DN: unavailable
>>>>> 2025-10-14 15:06:51,866 ERROR
>>>>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007:
>> Authorization
>>>>> failure sending will message: AMQ229031: Unable to validate user from
>>>>> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN:
>>>>> unavailable
>>>>>
>>>>> Regards,
>>>>> Paul Shields
>>>>>
>>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
>> For additional commands, e-mail: users-help@activemq.apache.org
>> For further information, visit: https://activemq.apache.org/contact
>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: MQTT Last Will not sent because denied authentication
Posted by Justin Bertram <jb...@apache.org>.
I just granted your Jira account request.
Justin
On Tue, Oct 28, 2025 at 2:53 PM Shields, Paul <pa...@hpe.com.invalid>
wrote:
> I don’t have a ASF Jira account. I am happy to request one to open a Jira
> ticket for this.
>
> Paul
>
> From: Justin Bertram <jb...@apache.org>
> Date: Tuesday, October 28, 2025 at 2:20 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: MQTT Last Will not sent because denied authentication
>
> I tend to agree with Matt here, but I think perhaps a feature could be
> added to Artemis so that LWT are authorized on connect to avoid this kind
> of problem. This behavior would be off by default so as not to impact
> existing users.
>
> Do you have a Jira account? If so, would you mind opening an issue [1] for
> this?
>
>
> Justin
>
> [1]
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/ARTEMIS__;!!NpxR!hdQbIYBgb_bBdoWH7TVo30l1yIKlErqg_yZOBlMZLt8SWQyPYex2pjuUSytWaI1zd5pSClzgpi0oxTF0wA$
>
> On Tue, Oct 28, 2025 at 1:58 PM Matt Pavlovich <ma...@apache.org>
> wrote:
>
> > Hi Paul-
> >
> > The usage of JWT and LWT are competing features, since JWT expires and
> LWT
> > is intended to alert for unplanned disconnect of long-running
> connections.
> > A possible solution is to perform periodic close-open of the connections
> > and re-register the LWT send would always fall within the timeframe of
> the
> > JWT expiry.
> >
> > Matt Pavlovich
> >
> > > On Oct 28, 2025, at 11:36 AM, Shields, Paul
> <pa...@hpe.com.INVALID>
> > wrote:
> > >
> > > Hi Justin,
> > >
> > > I am struggling with this statement
> > > However, authorization for actually
> > > sending the will message is not performed at this point. Authorization
> is
> > > only performed when the LWT message is actually sent (e.g. when the
> > > client's connection fails). All sending operations are authorized at
> the
> > > time they occur whether that's via the normal PUBLISH packet from the
> > > client or for a LWT message sent on the client's behalf by the broker.
> > > If we were using a user name and password (even if that was stored in
> > LDAP) that would be ok. But we are using the machine ID for the user name
> > and a JWT is used in the password field of the MQTT connect. My problem
> is
> > that the JWTs for authentication have a short expiration time (5min).
> This
> > link explains the JWT concept we are using for authentication.
> >
> https://urldefense.com/v3/__https://stytch.com/blog/understanding-jwks/__;!!NpxR!hdQbIYBgb_bBdoWH7TVo30l1yIKlErqg_yZOBlMZLt8SWQyPYex2pjuUSytWaI1zd5pSClzgpi1KjsUsCg$
> but instead of using Stytch
> > as the authorization server JWKS https endpoint we are using Spire with a
> > plugin for the JWKS. The specific issue is that the JWT supplied with the
> > connect to the broker has expired by the time the LWT is being delivered.
> > We chose the MQTT protocol for its simplicity and the LWT feature for
> > server heartbeat.
> > >
> > > Not sure we have implemented the authorize function of the
> > securityManager class in alignment with how Artemis operates. We do not
> > create users up front but on the fly as each machine connects to publish
> > its status or scribe to another's status. We use the admin role for all
> of
> > the machine users/IDs. One on of the key features for using the JWT for
> us
> > is that we imbed the machine ID in the jwt payload and in the authorize
> > function reject any publish (SEND) request if the target topic does not
> > match the sender machine ID. Not sure how we would update the JWT for
> every
> > connection every 5min so that the JWT would be valid at the time it will
> be
> > sent?
> > >
> > > Is there a reference implementation of a pluggable securityManager that
> > uses JWTs?
> > >
> > > Any other suggestions?
> > >
> > > Regards,
> > > Paul
> > >
> > > From: Justin Bertram <jb...@apache.org>
> > > Date: Tuesday, October 28, 2025 at 9:34 AM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: MQTT Last Will not sent because denied authentication
> > >
> > > Just following up to ensure my explanation made sense. Do you need
> > anything
> > > further here?
> > >
> > >
> > > Justin
> > >
> > > On Fri, Oct 24, 2025 at 4:17 PM Justin Bertram <jb...@apache.org>
> > wrote:
> > >
> > >> An MQTT client connects to the broker via a CONNECT packet. Typically
> > this
> > >> packet contains the client's credentials which are then authenticated
> by
> > >> the broker.
> > >>
> > >> Keep in mind that authentication and authorization are related but
> > >> separate things. A client may be authenticated but not authorized to
> > >> consume messages from or send messages to specific topics. This is a
> > >> fundamental concept of role-based access control.
> > >>
> > >> The CONNECT packet also contains all the details about the LWT message
> > >> (i.e. payload, properties, & topic). However, authorization for
> actually
> > >> sending the will message is not performed at this point. Authorization
> > is
> > >> only performed when the LWT message is actually sent (e.g. when the
> > >> client's connection fails). All sending operations are authorized at
> the
> > >> time they occur whether that's via the normal PUBLISH packet from the
> > >> client or for a LWT message sent on the client's behalf by the broker.
> > >>
> > >> Authorization is important here because the destination topic for the
> > LWT
> > >> message is arbitrary. If authorization was not performed then it would
> > be
> > >> simple for a client to send a message to a topic which it would not
> > >> otherwise have authorization.
> > >>
> > >> If authorization is failing when the broker attempts to send the
> > client's
> > >> LWT message and you're circumventing this so that the message is
> > actually
> > >> sent then it seems you may be undermining the security of your
> > environment.
> > >> You may inadvertently allow a clever, nefarious actor to send a
> message
> > to
> > >> a topic to which they are not authorized.
> > >>
> > >> How are you currently re-authenticating your clients when their JWTs
> > >> expire?
> > >>
> > >>
> > >> Justin
> > >>
> > >> On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul
> > <pa...@hpe.com.invalid>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Why are MQTT last will messages authenticated with Artemis before
> being
> > >>> sent? From my understanding of the MQTT last will feature is that
> > >>> authentication is a separate step handled during the client's initial
> > >>> connection. The LWT itself is a message prepared by a client and
> > stored by
> > >>> the broker to be published only if the client disconnects
> > unexpectedly. The
> > >>> security of the LWT message is therefore dependent on the security of
> > the
> > >>> initial client connection, which must be established through methods
> > like
> > >>> username/password or TLS and not when the LWT is delivered/published.
> > >>>
> > >>> We have written a custom securityManager plugin that uses Json Web
> > >>> Tokens as passwords for connecting MQTT clients. We use MQTT for
> server
> > >>> availability which has a state of either up or down. When a server
> > client
> > >>> connects with the MQTT Broker it is authenticated with the broker
> > using a
> > >>> JWT, registers a last will, and then publishes a server up message
> on a
> > >>> MQTT topic. We have other MQTT clients that also connect with the
> > Artemis
> > >>> broker using JWTs for auth and then subscribe to server state topics.
> > The
> > >>> MQTT last will is used to publish server down messages when the long
> > >>> running server dies for some reason. Some of the servers publish
> > >>> “re-announce” messages as they complete certain steps of processing
> > that
> > >>> they publish on a different topic using the initial client
> connection.
> > >>> This was initially developed using Artemis 2.28.0 and later used in
> > Artemis
> > >>> 2.34.0 and 2.40.0. Our securityManager plugin,
> JwtJassSecurityManager,
> > >>> implements the ActiveMQSecurityManager5 interface. During our initial
> > >>> development of the securityManager plugin we saw MQTT last will
> > messages
> > >>> failing authentication and put a workaround in place to check that if
> > there
> > >>> is a subject and a principal with the same user name (each MQTT
> client
> > has
> > >>> a unique user name) which means that this is the same session that
> has
> > >>> previously been authenticated, we will validate the JWT, but ignore
> the
> > >>> expiration time. We are now upgrading to Artemis 2.43.0 and would
> > like to
> > >>> remove the workaround. Our workaround for last will authentication
> > breaks
> > >>> down when the keys are rotated. It is about to become a real problem
> > as we
> > >>> are going to rotate the JWT key on a more frequent schedule. Here are
> > the
> > >>> errors produced:
> > >>>
> > >>> 2025-10-14 15:06:51,858 INFO
> > >>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed:
> > username:
> > >>> x3000c0s9b0n0, details: auth is invalid
> > >>> InvalidJwtException: JWT processing failed. Additional details: [[17]
> > >>> Unable to process JOSE object (cause:
> > >>> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
> > >>> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
> > >>> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
> > >>> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
> > >>> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256,
> n=CENSORED,
> > >>> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C,
> > alg=RS256,
> > >>> n=CENSORED, e=CENSORED}]):
> > JsonWebSignature{"alg":"RS256","kid”:”previous
> > >>> KEY-A","typ":”JWT”CENSORED
> > >>> ]
> > >>> 2025-10-14 15:06:51,861 INFO
> > >>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication
> > for
> > >>> user: x3000c0s9b0n0, subject: Subject:
> > >>> Principal: pzxQ2XQN
> > >>> Principal: admin
> > >>> Principal: x3000c0s9b0n0
> > >>>
> > >>> 2025-10-14 15:06:51,863 WARN
> [org.apache.activemq.artemis.core.server]
> > >>> AMQ222216: Security problem while authenticating: AMQ229031: Unable
> to
> > >>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
> > >>> certificate subject DN: unavailable
> > >>> 2025-10-14 15:06:51,863 ERROR
> > >>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
> > >>> processing control packet:
> > >>> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
> > >>> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true,
> remainingLength=67],
> > >>>
> >
> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
> > >>> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap:
> > 27/27,
> > >>> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
> > >>> org.apache.activemq.artemis.api.core.ActiveMQSecurityException:
> > >>> AMQ229031: Unable to validate user from 127.0.0.6:49859. Username:
> > >>> x3000c0s9b0n0; SSL certificate subject DN: unavailable
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
> > >>> at
> > >>> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
> > >>> at
> > >>>
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> > >>> at
> > >>>
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> > >>> at
> > >>>
> >
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
> > >>> 2025-10-14 15:06:51,866 WARN
> [org.apache.activemq.artemis.core.server]
> > >>> AMQ222216: Security problem while authenticating: AMQ229031: Unable
> to
> > >>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
> > >>> certificate subject DN: unavailable
> > >>> 2025-10-14 15:06:51,866 ERROR
> > >>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007:
> > Authorization
> > >>> failure sending will message: AMQ229031: Unable to validate user from
> > >>> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject
> DN:
> > >>> unavailable
> > >>>
> > >>> Regards,
> > >>> Paul Shields
> > >>>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org
> > For further information, visit:
> https://urldefense.com/v3/__https://activemq.apache.org/contact__;!!NpxR!hdQbIYBgb_bBdoWH7TVo30l1yIKlErqg_yZOBlMZLt8SWQyPYex2pjuUSytWaI1zd5pSClzgpi1wpKbyCQ$
> >
> >
> >
>
Re: MQTT Last Will not sent because denied authentication
Posted by "Shields, Paul" <pa...@hpe.com.INVALID>.
I don’t have a ASF Jira account. I am happy to request one to open a Jira ticket for this.
Paul
From: Justin Bertram <jb...@apache.org>
Date: Tuesday, October 28, 2025 at 2:20 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: MQTT Last Will not sent because denied authentication
I tend to agree with Matt here, but I think perhaps a feature could be
added to Artemis so that LWT are authorized on connect to avoid this kind
of problem. This behavior would be off by default so as not to impact
existing users.
Do you have a Jira account? If so, would you mind opening an issue [1] for
this?
Justin
[1] https://urldefense.com/v3/__https://issues.apache.org/jira/browse/ARTEMIS__;!!NpxR!hdQbIYBgb_bBdoWH7TVo30l1yIKlErqg_yZOBlMZLt8SWQyPYex2pjuUSytWaI1zd5pSClzgpi0oxTF0wA$
On Tue, Oct 28, 2025 at 1:58 PM Matt Pavlovich <ma...@apache.org> wrote:
> Hi Paul-
>
> The usage of JWT and LWT are competing features, since JWT expires and LWT
> is intended to alert for unplanned disconnect of long-running connections.
> A possible solution is to perform periodic close-open of the connections
> and re-register the LWT send would always fall within the timeframe of the
> JWT expiry.
>
> Matt Pavlovich
>
> > On Oct 28, 2025, at 11:36 AM, Shields, Paul <pa...@hpe.com.INVALID>
> wrote:
> >
> > Hi Justin,
> >
> > I am struggling with this statement
> > However, authorization for actually
> > sending the will message is not performed at this point. Authorization is
> > only performed when the LWT message is actually sent (e.g. when the
> > client's connection fails). All sending operations are authorized at the
> > time they occur whether that's via the normal PUBLISH packet from the
> > client or for a LWT message sent on the client's behalf by the broker.
> > If we were using a user name and password (even if that was stored in
> LDAP) that would be ok. But we are using the machine ID for the user name
> and a JWT is used in the password field of the MQTT connect. My problem is
> that the JWTs for authentication have a short expiration time (5min). This
> link explains the JWT concept we are using for authentication.
> https://urldefense.com/v3/__https://stytch.com/blog/understanding-jwks/__;!!NpxR!hdQbIYBgb_bBdoWH7TVo30l1yIKlErqg_yZOBlMZLt8SWQyPYex2pjuUSytWaI1zd5pSClzgpi1KjsUsCg$ but instead of using Stytch
> as the authorization server JWKS https endpoint we are using Spire with a
> plugin for the JWKS. The specific issue is that the JWT supplied with the
> connect to the broker has expired by the time the LWT is being delivered.
> We chose the MQTT protocol for its simplicity and the LWT feature for
> server heartbeat.
> >
> > Not sure we have implemented the authorize function of the
> securityManager class in alignment with how Artemis operates. We do not
> create users up front but on the fly as each machine connects to publish
> its status or scribe to another's status. We use the admin role for all of
> the machine users/IDs. One on of the key features for using the JWT for us
> is that we imbed the machine ID in the jwt payload and in the authorize
> function reject any publish (SEND) request if the target topic does not
> match the sender machine ID. Not sure how we would update the JWT for every
> connection every 5min so that the JWT would be valid at the time it will be
> sent?
> >
> > Is there a reference implementation of a pluggable securityManager that
> uses JWTs?
> >
> > Any other suggestions?
> >
> > Regards,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>
> > Date: Tuesday, October 28, 2025 at 9:34 AM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: MQTT Last Will not sent because denied authentication
> >
> > Just following up to ensure my explanation made sense. Do you need
> anything
> > further here?
> >
> >
> > Justin
> >
> > On Fri, Oct 24, 2025 at 4:17 PM Justin Bertram <jb...@apache.org>
> wrote:
> >
> >> An MQTT client connects to the broker via a CONNECT packet. Typically
> this
> >> packet contains the client's credentials which are then authenticated by
> >> the broker.
> >>
> >> Keep in mind that authentication and authorization are related but
> >> separate things. A client may be authenticated but not authorized to
> >> consume messages from or send messages to specific topics. This is a
> >> fundamental concept of role-based access control.
> >>
> >> The CONNECT packet also contains all the details about the LWT message
> >> (i.e. payload, properties, & topic). However, authorization for actually
> >> sending the will message is not performed at this point. Authorization
> is
> >> only performed when the LWT message is actually sent (e.g. when the
> >> client's connection fails). All sending operations are authorized at the
> >> time they occur whether that's via the normal PUBLISH packet from the
> >> client or for a LWT message sent on the client's behalf by the broker.
> >>
> >> Authorization is important here because the destination topic for the
> LWT
> >> message is arbitrary. If authorization was not performed then it would
> be
> >> simple for a client to send a message to a topic which it would not
> >> otherwise have authorization.
> >>
> >> If authorization is failing when the broker attempts to send the
> client's
> >> LWT message and you're circumventing this so that the message is
> actually
> >> sent then it seems you may be undermining the security of your
> environment.
> >> You may inadvertently allow a clever, nefarious actor to send a message
> to
> >> a topic to which they are not authorized.
> >>
> >> How are you currently re-authenticating your clients when their JWTs
> >> expire?
> >>
> >>
> >> Justin
> >>
> >> On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul
> <pa...@hpe.com.invalid>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Why are MQTT last will messages authenticated with Artemis before being
> >>> sent? From my understanding of the MQTT last will feature is that
> >>> authentication is a separate step handled during the client's initial
> >>> connection. The LWT itself is a message prepared by a client and
> stored by
> >>> the broker to be published only if the client disconnects
> unexpectedly. The
> >>> security of the LWT message is therefore dependent on the security of
> the
> >>> initial client connection, which must be established through methods
> like
> >>> username/password or TLS and not when the LWT is delivered/published.
> >>>
> >>> We have written a custom securityManager plugin that uses Json Web
> >>> Tokens as passwords for connecting MQTT clients. We use MQTT for server
> >>> availability which has a state of either up or down. When a server
> client
> >>> connects with the MQTT Broker it is authenticated with the broker
> using a
> >>> JWT, registers a last will, and then publishes a server up message on a
> >>> MQTT topic. We have other MQTT clients that also connect with the
> Artemis
> >>> broker using JWTs for auth and then subscribe to server state topics.
> The
> >>> MQTT last will is used to publish server down messages when the long
> >>> running server dies for some reason. Some of the servers publish
> >>> “re-announce” messages as they complete certain steps of processing
> that
> >>> they publish on a different topic using the initial client connection.
> >>> This was initially developed using Artemis 2.28.0 and later used in
> Artemis
> >>> 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager,
> >>> implements the ActiveMQSecurityManager5 interface. During our initial
> >>> development of the securityManager plugin we saw MQTT last will
> messages
> >>> failing authentication and put a workaround in place to check that if
> there
> >>> is a subject and a principal with the same user name (each MQTT client
> has
> >>> a unique user name) which means that this is the same session that has
> >>> previously been authenticated, we will validate the JWT, but ignore the
> >>> expiration time. We are now upgrading to Artemis 2.43.0 and would
> like to
> >>> remove the workaround. Our workaround for last will authentication
> breaks
> >>> down when the keys are rotated. It is about to become a real problem
> as we
> >>> are going to rotate the JWT key on a more frequent schedule. Here are
> the
> >>> errors produced:
> >>>
> >>> 2025-10-14 15:06:51,858 INFO
> >>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed:
> username:
> >>> x3000c0s9b0n0, details: auth is invalid
> >>> InvalidJwtException: JWT processing failed. Additional details: [[17]
> >>> Unable to process JOSE object (cause:
> >>> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
> >>> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
> >>> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
> >>> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
> >>> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED,
> >>> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C,
> alg=RS256,
> >>> n=CENSORED, e=CENSORED}]):
> JsonWebSignature{"alg":"RS256","kid”:”previous
> >>> KEY-A","typ":”JWT”CENSORED
> >>> ]
> >>> 2025-10-14 15:06:51,861 INFO
> >>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication
> for
> >>> user: x3000c0s9b0n0, subject: Subject:
> >>> Principal: pzxQ2XQN
> >>> Principal: admin
> >>> Principal: x3000c0s9b0n0
> >>>
> >>> 2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server]
> >>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
> >>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
> >>> certificate subject DN: unavailable
> >>> 2025-10-14 15:06:51,863 ERROR
> >>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
> >>> processing control packet:
> >>> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
> >>> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67],
> >>>
> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
> >>> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap:
> 27/27,
> >>> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
> >>> org.apache.activemq.artemis.api.core.ActiveMQSecurityException:
> >>> AMQ229031: Unable to validate user from 127.0.0.6:49859. Username:
> >>> x3000c0s9b0n0; SSL certificate subject DN: unavailable
> >>> at
> >>>
> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
> >>> at
> >>>
> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
> >>> at
> >>>
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
> >>> at
> >>>
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
> >>> at
> >>>
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
> >>> at
> >>>
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
> >>> at
> >>>
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
> >>> at
> >>>
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
> >>> at
> >>>
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
> >>> at
> >>> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
> >>> at
> >>>
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
> >>> at
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> >>> at
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> >>> at
> >>>
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
> >>> 2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server]
> >>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
> >>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
> >>> certificate subject DN: unavailable
> >>> 2025-10-14 15:06:51,866 ERROR
> >>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007:
> Authorization
> >>> failure sending will message: AMQ229031: Unable to validate user from
> >>> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN:
> >>> unavailable
> >>>
> >>> Regards,
> >>> Paul Shields
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://urldefense.com/v3/__https://activemq.apache.org/contact__;!!NpxR!hdQbIYBgb_bBdoWH7TVo30l1yIKlErqg_yZOBlMZLt8SWQyPYex2pjuUSytWaI1zd5pSClzgpi1wpKbyCQ$
>
>
>
Re: MQTT Last Will not sent because denied authentication
Posted by Justin Bertram <jb...@apache.org>.
I tend to agree with Matt here, but I think perhaps a feature could be
added to Artemis so that LWT are authorized on connect to avoid this kind
of problem. This behavior would be off by default so as not to impact
existing users.
Do you have a Jira account? If so, would you mind opening an issue [1] for
this?
Justin
[1] https://issues.apache.org/jira/browse/ARTEMIS
On Tue, Oct 28, 2025 at 1:58 PM Matt Pavlovich <ma...@apache.org> wrote:
> Hi Paul-
>
> The usage of JWT and LWT are competing features, since JWT expires and LWT
> is intended to alert for unplanned disconnect of long-running connections.
> A possible solution is to perform periodic close-open of the connections
> and re-register the LWT send would always fall within the timeframe of the
> JWT expiry.
>
> Matt Pavlovich
>
> > On Oct 28, 2025, at 11:36 AM, Shields, Paul <pa...@hpe.com.INVALID>
> wrote:
> >
> > Hi Justin,
> >
> > I am struggling with this statement
> > However, authorization for actually
> > sending the will message is not performed at this point. Authorization is
> > only performed when the LWT message is actually sent (e.g. when the
> > client's connection fails). All sending operations are authorized at the
> > time they occur whether that's via the normal PUBLISH packet from the
> > client or for a LWT message sent on the client's behalf by the broker.
> > If we were using a user name and password (even if that was stored in
> LDAP) that would be ok. But we are using the machine ID for the user name
> and a JWT is used in the password field of the MQTT connect. My problem is
> that the JWTs for authentication have a short expiration time (5min). This
> link explains the JWT concept we are using for authentication.
> https://stytch.com/blog/understanding-jwks/ but instead of using Stytch
> as the authorization server JWKS https endpoint we are using Spire with a
> plugin for the JWKS. The specific issue is that the JWT supplied with the
> connect to the broker has expired by the time the LWT is being delivered.
> We chose the MQTT protocol for its simplicity and the LWT feature for
> server heartbeat.
> >
> > Not sure we have implemented the authorize function of the
> securityManager class in alignment with how Artemis operates. We do not
> create users up front but on the fly as each machine connects to publish
> its status or scribe to another's status. We use the admin role for all of
> the machine users/IDs. One on of the key features for using the JWT for us
> is that we imbed the machine ID in the jwt payload and in the authorize
> function reject any publish (SEND) request if the target topic does not
> match the sender machine ID. Not sure how we would update the JWT for every
> connection every 5min so that the JWT would be valid at the time it will be
> sent?
> >
> > Is there a reference implementation of a pluggable securityManager that
> uses JWTs?
> >
> > Any other suggestions?
> >
> > Regards,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>
> > Date: Tuesday, October 28, 2025 at 9:34 AM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: MQTT Last Will not sent because denied authentication
> >
> > Just following up to ensure my explanation made sense. Do you need
> anything
> > further here?
> >
> >
> > Justin
> >
> > On Fri, Oct 24, 2025 at 4:17 PM Justin Bertram <jb...@apache.org>
> wrote:
> >
> >> An MQTT client connects to the broker via a CONNECT packet. Typically
> this
> >> packet contains the client's credentials which are then authenticated by
> >> the broker.
> >>
> >> Keep in mind that authentication and authorization are related but
> >> separate things. A client may be authenticated but not authorized to
> >> consume messages from or send messages to specific topics. This is a
> >> fundamental concept of role-based access control.
> >>
> >> The CONNECT packet also contains all the details about the LWT message
> >> (i.e. payload, properties, & topic). However, authorization for actually
> >> sending the will message is not performed at this point. Authorization
> is
> >> only performed when the LWT message is actually sent (e.g. when the
> >> client's connection fails). All sending operations are authorized at the
> >> time they occur whether that's via the normal PUBLISH packet from the
> >> client or for a LWT message sent on the client's behalf by the broker.
> >>
> >> Authorization is important here because the destination topic for the
> LWT
> >> message is arbitrary. If authorization was not performed then it would
> be
> >> simple for a client to send a message to a topic which it would not
> >> otherwise have authorization.
> >>
> >> If authorization is failing when the broker attempts to send the
> client's
> >> LWT message and you're circumventing this so that the message is
> actually
> >> sent then it seems you may be undermining the security of your
> environment.
> >> You may inadvertently allow a clever, nefarious actor to send a message
> to
> >> a topic to which they are not authorized.
> >>
> >> How are you currently re-authenticating your clients when their JWTs
> >> expire?
> >>
> >>
> >> Justin
> >>
> >> On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul
> <pa...@hpe.com.invalid>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Why are MQTT last will messages authenticated with Artemis before being
> >>> sent? From my understanding of the MQTT last will feature is that
> >>> authentication is a separate step handled during the client's initial
> >>> connection. The LWT itself is a message prepared by a client and
> stored by
> >>> the broker to be published only if the client disconnects
> unexpectedly. The
> >>> security of the LWT message is therefore dependent on the security of
> the
> >>> initial client connection, which must be established through methods
> like
> >>> username/password or TLS and not when the LWT is delivered/published.
> >>>
> >>> We have written a custom securityManager plugin that uses Json Web
> >>> Tokens as passwords for connecting MQTT clients. We use MQTT for server
> >>> availability which has a state of either up or down. When a server
> client
> >>> connects with the MQTT Broker it is authenticated with the broker
> using a
> >>> JWT, registers a last will, and then publishes a server up message on a
> >>> MQTT topic. We have other MQTT clients that also connect with the
> Artemis
> >>> broker using JWTs for auth and then subscribe to server state topics.
> The
> >>> MQTT last will is used to publish server down messages when the long
> >>> running server dies for some reason. Some of the servers publish
> >>> “re-announce” messages as they complete certain steps of processing
> that
> >>> they publish on a different topic using the initial client connection.
> >>> This was initially developed using Artemis 2.28.0 and later used in
> Artemis
> >>> 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager,
> >>> implements the ActiveMQSecurityManager5 interface. During our initial
> >>> development of the securityManager plugin we saw MQTT last will
> messages
> >>> failing authentication and put a workaround in place to check that if
> there
> >>> is a subject and a principal with the same user name (each MQTT client
> has
> >>> a unique user name) which means that this is the same session that has
> >>> previously been authenticated, we will validate the JWT, but ignore the
> >>> expiration time. We are now upgrading to Artemis 2.43.0 and would
> like to
> >>> remove the workaround. Our workaround for last will authentication
> breaks
> >>> down when the keys are rotated. It is about to become a real problem
> as we
> >>> are going to rotate the JWT key on a more frequent schedule. Here are
> the
> >>> errors produced:
> >>>
> >>> 2025-10-14 15:06:51,858 INFO
> >>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed:
> username:
> >>> x3000c0s9b0n0, details: auth is invalid
> >>> InvalidJwtException: JWT processing failed. Additional details: [[17]
> >>> Unable to process JOSE object (cause:
> >>> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
> >>> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
> >>> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
> >>> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
> >>> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED,
> >>> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C,
> alg=RS256,
> >>> n=CENSORED, e=CENSORED}]):
> JsonWebSignature{"alg":"RS256","kid”:”previous
> >>> KEY-A","typ":”JWT”CENSORED
> >>> ]
> >>> 2025-10-14 15:06:51,861 INFO
> >>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication
> for
> >>> user: x3000c0s9b0n0, subject: Subject:
> >>> Principal: pzxQ2XQN
> >>> Principal: admin
> >>> Principal: x3000c0s9b0n0
> >>>
> >>> 2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server]
> >>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
> >>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
> >>> certificate subject DN: unavailable
> >>> 2025-10-14 15:06:51,863 ERROR
> >>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
> >>> processing control packet:
> >>> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
> >>> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67],
> >>>
> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
> >>> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap:
> 27/27,
> >>> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
> >>> org.apache.activemq.artemis.api.core.ActiveMQSecurityException:
> >>> AMQ229031: Unable to validate user from 127.0.0.6:49859. Username:
> >>> x3000c0s9b0n0; SSL certificate subject DN: unavailable
> >>> at
> >>>
> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
> >>> at
> >>>
> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
> >>> at
> >>>
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
> >>> at
> >>>
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
> >>> at
> >>>
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
> >>> at
> >>>
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
> >>> at
> >>>
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
> >>> at
> >>>
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
> >>> at
> >>>
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
> >>> at
> >>> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
> >>> at
> >>>
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
> >>> at
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> >>> at
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> >>> at
> >>>
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
> >>> 2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server]
> >>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
> >>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
> >>> certificate subject DN: unavailable
> >>> 2025-10-14 15:06:51,866 ERROR
> >>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007:
> Authorization
> >>> failure sending will message: AMQ229031: Unable to validate user from
> >>> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN:
> >>> unavailable
> >>>
> >>> Regards,
> >>> Paul Shields
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
>
Re: MQTT Last Will not sent because denied authentication
Posted by Matt Pavlovich <ma...@apache.org>.
Hi Paul-
The usage of JWT and LWT are competing features, since JWT expires and LWT is intended to alert for unplanned disconnect of long-running connections. A possible solution is to perform periodic close-open of the connections and re-register the LWT send would always fall within the timeframe of the JWT expiry.
Matt Pavlovich
> On Oct 28, 2025, at 11:36 AM, Shields, Paul <pa...@hpe.com.INVALID> wrote:
>
> Hi Justin,
>
> I am struggling with this statement
> However, authorization for actually
> sending the will message is not performed at this point. Authorization is
> only performed when the LWT message is actually sent (e.g. when the
> client's connection fails). All sending operations are authorized at the
> time they occur whether that's via the normal PUBLISH packet from the
> client or for a LWT message sent on the client's behalf by the broker.
> If we were using a user name and password (even if that was stored in LDAP) that would be ok. But we are using the machine ID for the user name and a JWT is used in the password field of the MQTT connect. My problem is that the JWTs for authentication have a short expiration time (5min). This link explains the JWT concept we are using for authentication. https://stytch.com/blog/understanding-jwks/ but instead of using Stytch as the authorization server JWKS https endpoint we are using Spire with a plugin for the JWKS. The specific issue is that the JWT supplied with the connect to the broker has expired by the time the LWT is being delivered. We chose the MQTT protocol for its simplicity and the LWT feature for server heartbeat.
>
> Not sure we have implemented the authorize function of the securityManager class in alignment with how Artemis operates. We do not create users up front but on the fly as each machine connects to publish its status or scribe to another's status. We use the admin role for all of the machine users/IDs. One on of the key features for using the JWT for us is that we imbed the machine ID in the jwt payload and in the authorize function reject any publish (SEND) request if the target topic does not match the sender machine ID. Not sure how we would update the JWT for every connection every 5min so that the JWT would be valid at the time it will be sent?
>
> Is there a reference implementation of a pluggable securityManager that uses JWTs?
>
> Any other suggestions?
>
> Regards,
> Paul
>
> From: Justin Bertram <jb...@apache.org>
> Date: Tuesday, October 28, 2025 at 9:34 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: MQTT Last Will not sent because denied authentication
>
> Just following up to ensure my explanation made sense. Do you need anything
> further here?
>
>
> Justin
>
> On Fri, Oct 24, 2025 at 4:17 PM Justin Bertram <jb...@apache.org> wrote:
>
>> An MQTT client connects to the broker via a CONNECT packet. Typically this
>> packet contains the client's credentials which are then authenticated by
>> the broker.
>>
>> Keep in mind that authentication and authorization are related but
>> separate things. A client may be authenticated but not authorized to
>> consume messages from or send messages to specific topics. This is a
>> fundamental concept of role-based access control.
>>
>> The CONNECT packet also contains all the details about the LWT message
>> (i.e. payload, properties, & topic). However, authorization for actually
>> sending the will message is not performed at this point. Authorization is
>> only performed when the LWT message is actually sent (e.g. when the
>> client's connection fails). All sending operations are authorized at the
>> time they occur whether that's via the normal PUBLISH packet from the
>> client or for a LWT message sent on the client's behalf by the broker.
>>
>> Authorization is important here because the destination topic for the LWT
>> message is arbitrary. If authorization was not performed then it would be
>> simple for a client to send a message to a topic which it would not
>> otherwise have authorization.
>>
>> If authorization is failing when the broker attempts to send the client's
>> LWT message and you're circumventing this so that the message is actually
>> sent then it seems you may be undermining the security of your environment.
>> You may inadvertently allow a clever, nefarious actor to send a message to
>> a topic to which they are not authorized.
>>
>> How are you currently re-authenticating your clients when their JWTs
>> expire?
>>
>>
>> Justin
>>
>> On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul <pa...@hpe.com.invalid>
>> wrote:
>>
>>> Hi,
>>>
>>> Why are MQTT last will messages authenticated with Artemis before being
>>> sent? From my understanding of the MQTT last will feature is that
>>> authentication is a separate step handled during the client's initial
>>> connection. The LWT itself is a message prepared by a client and stored by
>>> the broker to be published only if the client disconnects unexpectedly. The
>>> security of the LWT message is therefore dependent on the security of the
>>> initial client connection, which must be established through methods like
>>> username/password or TLS and not when the LWT is delivered/published.
>>>
>>> We have written a custom securityManager plugin that uses Json Web
>>> Tokens as passwords for connecting MQTT clients. We use MQTT for server
>>> availability which has a state of either up or down. When a server client
>>> connects with the MQTT Broker it is authenticated with the broker using a
>>> JWT, registers a last will, and then publishes a server up message on a
>>> MQTT topic. We have other MQTT clients that also connect with the Artemis
>>> broker using JWTs for auth and then subscribe to server state topics. The
>>> MQTT last will is used to publish server down messages when the long
>>> running server dies for some reason. Some of the servers publish
>>> “re-announce” messages as they complete certain steps of processing that
>>> they publish on a different topic using the initial client connection.
>>> This was initially developed using Artemis 2.28.0 and later used in Artemis
>>> 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager,
>>> implements the ActiveMQSecurityManager5 interface. During our initial
>>> development of the securityManager plugin we saw MQTT last will messages
>>> failing authentication and put a workaround in place to check that if there
>>> is a subject and a principal with the same user name (each MQTT client has
>>> a unique user name) which means that this is the same session that has
>>> previously been authenticated, we will validate the JWT, but ignore the
>>> expiration time. We are now upgrading to Artemis 2.43.0 and would like to
>>> remove the workaround. Our workaround for last will authentication breaks
>>> down when the keys are rotated. It is about to become a real problem as we
>>> are going to rotate the JWT key on a more frequent schedule. Here are the
>>> errors produced:
>>>
>>> 2025-10-14 15:06:51,858 INFO
>>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed: username:
>>> x3000c0s9b0n0, details: auth is invalid
>>> InvalidJwtException: JWT processing failed. Additional details: [[17]
>>> Unable to process JOSE object (cause:
>>> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
>>> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
>>> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
>>> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
>>> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED,
>>> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C, alg=RS256,
>>> n=CENSORED, e=CENSORED}]): JsonWebSignature{"alg":"RS256","kid”:”previous
>>> KEY-A","typ":”JWT”CENSORED
>>> ]
>>> 2025-10-14 15:06:51,861 INFO
>>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication for
>>> user: x3000c0s9b0n0, subject: Subject:
>>> Principal: pzxQ2XQN
>>> Principal: admin
>>> Principal: x3000c0s9b0n0
>>>
>>> 2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server]
>>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>>> certificate subject DN: unavailable
>>> 2025-10-14 15:06:51,863 ERROR
>>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
>>> processing control packet:
>>> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
>>> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67],
>>> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
>>> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap: 27/27,
>>> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
>>> org.apache.activemq.artemis.api.core.ActiveMQSecurityException:
>>> AMQ229031: Unable to validate user from 127.0.0.6:49859. Username:
>>> x3000c0s9b0n0; SSL certificate subject DN: unavailable
>>> at
>>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
>>> at
>>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
>>> at
>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
>>> at
>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
>>> at
>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
>>> at
>>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
>>> at
>>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
>>> at
>>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
>>> at
>>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
>>> at
>>> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
>>> at
>>> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>>> at
>>> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
>>> 2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server]
>>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>>> certificate subject DN: unavailable
>>> 2025-10-14 15:06:51,866 ERROR
>>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007: Authorization
>>> failure sending will message: AMQ229031: Unable to validate user from
>>> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN:
>>> unavailable
>>>
>>> Regards,
>>> Paul Shields
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: MQTT Last Will not sent because denied authentication
Posted by "Shields, Paul" <pa...@hpe.com.INVALID>.
Hi Justin,
I am struggling with this statement
However, authorization for actually
sending the will message is not performed at this point. Authorization is
only performed when the LWT message is actually sent (e.g. when the
client's connection fails). All sending operations are authorized at the
time they occur whether that's via the normal PUBLISH packet from the
client or for a LWT message sent on the client's behalf by the broker.
If we were using a user name and password (even if that was stored in LDAP) that would be ok. But we are using the machine ID for the user name and a JWT is used in the password field of the MQTT connect. My problem is that the JWTs for authentication have a short expiration time (5min). This link explains the JWT concept we are using for authentication. https://stytch.com/blog/understanding-jwks/ but instead of using Stytch as the authorization server JWKS https endpoint we are using Spire with a plugin for the JWKS. The specific issue is that the JWT supplied with the connect to the broker has expired by the time the LWT is being delivered. We chose the MQTT protocol for its simplicity and the LWT feature for server heartbeat.
Not sure we have implemented the authorize function of the securityManager class in alignment with how Artemis operates. We do not create users up front but on the fly as each machine connects to publish its status or scribe to another's status. We use the admin role for all of the machine users/IDs. One on of the key features for using the JWT for us is that we imbed the machine ID in the jwt payload and in the authorize function reject any publish (SEND) request if the target topic does not match the sender machine ID. Not sure how we would update the JWT for every connection every 5min so that the JWT would be valid at the time it will be sent?
Is there a reference implementation of a pluggable securityManager that uses JWTs?
Any other suggestions?
Regards,
Paul
From: Justin Bertram <jb...@apache.org>
Date: Tuesday, October 28, 2025 at 9:34 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: MQTT Last Will not sent because denied authentication
Just following up to ensure my explanation made sense. Do you need anything
further here?
Justin
On Fri, Oct 24, 2025 at 4:17 PM Justin Bertram <jb...@apache.org> wrote:
> An MQTT client connects to the broker via a CONNECT packet. Typically this
> packet contains the client's credentials which are then authenticated by
> the broker.
>
> Keep in mind that authentication and authorization are related but
> separate things. A client may be authenticated but not authorized to
> consume messages from or send messages to specific topics. This is a
> fundamental concept of role-based access control.
>
> The CONNECT packet also contains all the details about the LWT message
> (i.e. payload, properties, & topic). However, authorization for actually
> sending the will message is not performed at this point. Authorization is
> only performed when the LWT message is actually sent (e.g. when the
> client's connection fails). All sending operations are authorized at the
> time they occur whether that's via the normal PUBLISH packet from the
> client or for a LWT message sent on the client's behalf by the broker.
>
> Authorization is important here because the destination topic for the LWT
> message is arbitrary. If authorization was not performed then it would be
> simple for a client to send a message to a topic which it would not
> otherwise have authorization.
>
> If authorization is failing when the broker attempts to send the client's
> LWT message and you're circumventing this so that the message is actually
> sent then it seems you may be undermining the security of your environment.
> You may inadvertently allow a clever, nefarious actor to send a message to
> a topic to which they are not authorized.
>
> How are you currently re-authenticating your clients when their JWTs
> expire?
>
>
> Justin
>
> On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul <pa...@hpe.com.invalid>
> wrote:
>
>> Hi,
>>
>> Why are MQTT last will messages authenticated with Artemis before being
>> sent? From my understanding of the MQTT last will feature is that
>> authentication is a separate step handled during the client's initial
>> connection. The LWT itself is a message prepared by a client and stored by
>> the broker to be published only if the client disconnects unexpectedly. The
>> security of the LWT message is therefore dependent on the security of the
>> initial client connection, which must be established through methods like
>> username/password or TLS and not when the LWT is delivered/published.
>>
>> We have written a custom securityManager plugin that uses Json Web
>> Tokens as passwords for connecting MQTT clients. We use MQTT for server
>> availability which has a state of either up or down. When a server client
>> connects with the MQTT Broker it is authenticated with the broker using a
>> JWT, registers a last will, and then publishes a server up message on a
>> MQTT topic. We have other MQTT clients that also connect with the Artemis
>> broker using JWTs for auth and then subscribe to server state topics. The
>> MQTT last will is used to publish server down messages when the long
>> running server dies for some reason. Some of the servers publish
>> “re-announce” messages as they complete certain steps of processing that
>> they publish on a different topic using the initial client connection.
>> This was initially developed using Artemis 2.28.0 and later used in Artemis
>> 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager,
>> implements the ActiveMQSecurityManager5 interface. During our initial
>> development of the securityManager plugin we saw MQTT last will messages
>> failing authentication and put a workaround in place to check that if there
>> is a subject and a principal with the same user name (each MQTT client has
>> a unique user name) which means that this is the same session that has
>> previously been authenticated, we will validate the JWT, but ignore the
>> expiration time. We are now upgrading to Artemis 2.43.0 and would like to
>> remove the workaround. Our workaround for last will authentication breaks
>> down when the keys are rotated. It is about to become a real problem as we
>> are going to rotate the JWT key on a more frequent schedule. Here are the
>> errors produced:
>>
>> 2025-10-14 15:06:51,858 INFO
>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed: username:
>> x3000c0s9b0n0, details: auth is invalid
>> InvalidJwtException: JWT processing failed. Additional details: [[17]
>> Unable to process JOSE object (cause:
>> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
>> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
>> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
>> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
>> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED,
>> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C, alg=RS256,
>> n=CENSORED, e=CENSORED}]): JsonWebSignature{"alg":"RS256","kid”:”previous
>> KEY-A","typ":”JWT”CENSORED
>> ]
>> 2025-10-14 15:06:51,861 INFO
>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication for
>> user: x3000c0s9b0n0, subject: Subject:
>> Principal: pzxQ2XQN
>> Principal: admin
>> Principal: x3000c0s9b0n0
>>
>> 2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server]
>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>> certificate subject DN: unavailable
>> 2025-10-14 15:06:51,863 ERROR
>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
>> processing control packet:
>> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
>> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67],
>> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
>> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap: 27/27,
>> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
>> org.apache.activemq.artemis.api.core.ActiveMQSecurityException:
>> AMQ229031: Unable to validate user from 127.0.0.6:49859. Username:
>> x3000c0s9b0n0; SSL certificate subject DN: unavailable
>> at
>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
>> at
>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
>> at
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
>> at
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
>> at
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
>> at
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
>> at
>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
>> at
>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
>> at
>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
>> at
>> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
>> at
>> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>> at
>> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
>> 2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server]
>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>> certificate subject DN: unavailable
>> 2025-10-14 15:06:51,866 ERROR
>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007: Authorization
>> failure sending will message: AMQ229031: Unable to validate user from
>> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN:
>> unavailable
>>
>> Regards,
>> Paul Shields
>>
>
Re: MQTT Last Will not sent because denied authentication
Posted by Justin Bertram <jb...@apache.org>.
Just following up to ensure my explanation made sense. Do you need anything
further here?
Justin
On Fri, Oct 24, 2025 at 4:17 PM Justin Bertram <jb...@apache.org> wrote:
> An MQTT client connects to the broker via a CONNECT packet. Typically this
> packet contains the client's credentials which are then authenticated by
> the broker.
>
> Keep in mind that authentication and authorization are related but
> separate things. A client may be authenticated but not authorized to
> consume messages from or send messages to specific topics. This is a
> fundamental concept of role-based access control.
>
> The CONNECT packet also contains all the details about the LWT message
> (i.e. payload, properties, & topic). However, authorization for actually
> sending the will message is not performed at this point. Authorization is
> only performed when the LWT message is actually sent (e.g. when the
> client's connection fails). All sending operations are authorized at the
> time they occur whether that's via the normal PUBLISH packet from the
> client or for a LWT message sent on the client's behalf by the broker.
>
> Authorization is important here because the destination topic for the LWT
> message is arbitrary. If authorization was not performed then it would be
> simple for a client to send a message to a topic which it would not
> otherwise have authorization.
>
> If authorization is failing when the broker attempts to send the client's
> LWT message and you're circumventing this so that the message is actually
> sent then it seems you may be undermining the security of your environment.
> You may inadvertently allow a clever, nefarious actor to send a message to
> a topic to which they are not authorized.
>
> How are you currently re-authenticating your clients when their JWTs
> expire?
>
>
> Justin
>
> On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul <pa...@hpe.com.invalid>
> wrote:
>
>> Hi,
>>
>> Why are MQTT last will messages authenticated with Artemis before being
>> sent? From my understanding of the MQTT last will feature is that
>> authentication is a separate step handled during the client's initial
>> connection. The LWT itself is a message prepared by a client and stored by
>> the broker to be published only if the client disconnects unexpectedly. The
>> security of the LWT message is therefore dependent on the security of the
>> initial client connection, which must be established through methods like
>> username/password or TLS and not when the LWT is delivered/published.
>>
>> We have written a custom securityManager plugin that uses Json Web
>> Tokens as passwords for connecting MQTT clients. We use MQTT for server
>> availability which has a state of either up or down. When a server client
>> connects with the MQTT Broker it is authenticated with the broker using a
>> JWT, registers a last will, and then publishes a server up message on a
>> MQTT topic. We have other MQTT clients that also connect with the Artemis
>> broker using JWTs for auth and then subscribe to server state topics. The
>> MQTT last will is used to publish server down messages when the long
>> running server dies for some reason. Some of the servers publish
>> “re-announce” messages as they complete certain steps of processing that
>> they publish on a different topic using the initial client connection.
>> This was initially developed using Artemis 2.28.0 and later used in Artemis
>> 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager,
>> implements the ActiveMQSecurityManager5 interface. During our initial
>> development of the securityManager plugin we saw MQTT last will messages
>> failing authentication and put a workaround in place to check that if there
>> is a subject and a principal with the same user name (each MQTT client has
>> a unique user name) which means that this is the same session that has
>> previously been authenticated, we will validate the JWT, but ignore the
>> expiration time. We are now upgrading to Artemis 2.43.0 and would like to
>> remove the workaround. Our workaround for last will authentication breaks
>> down when the keys are rotated. It is about to become a real problem as we
>> are going to rotate the JWT key on a more frequent schedule. Here are the
>> errors produced:
>>
>> 2025-10-14 15:06:51,858 INFO
>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed: username:
>> x3000c0s9b0n0, details: auth is invalid
>> InvalidJwtException: JWT processing failed. Additional details: [[17]
>> Unable to process JOSE object (cause:
>> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
>> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
>> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
>> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
>> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED,
>> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C, alg=RS256,
>> n=CENSORED, e=CENSORED}]): JsonWebSignature{"alg":"RS256","kid”:”previous
>> KEY-A","typ":”JWT”CENSORED
>> ]
>> 2025-10-14 15:06:51,861 INFO
>> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication for
>> user: x3000c0s9b0n0, subject: Subject:
>> Principal: pzxQ2XQN
>> Principal: admin
>> Principal: x3000c0s9b0n0
>>
>> 2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server]
>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>> certificate subject DN: unavailable
>> 2025-10-14 15:06:51,863 ERROR
>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
>> processing control packet:
>> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
>> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67],
>> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
>> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap: 27/27,
>> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
>> org.apache.activemq.artemis.api.core.ActiveMQSecurityException:
>> AMQ229031: Unable to validate user from 127.0.0.6:49859. Username:
>> x3000c0s9b0n0; SSL certificate subject DN: unavailable
>> at
>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
>> at
>> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
>> at
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
>> at
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
>> at
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
>> at
>> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
>> at
>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
>> at
>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
>> at
>> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
>> at
>> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
>> at
>> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>> at
>> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
>> 2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server]
>> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
>> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
>> certificate subject DN: unavailable
>> 2025-10-14 15:06:51,866 ERROR
>> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007: Authorization
>> failure sending will message: AMQ229031: Unable to validate user from
>> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN:
>> unavailable
>>
>> Regards,
>> Paul Shields
>>
>
Re: MQTT Last Will not sent because denied authentication
Posted by Justin Bertram <jb...@apache.org>.
An MQTT client connects to the broker via a CONNECT packet. Typically this
packet contains the client's credentials which are then authenticated by
the broker.
Keep in mind that authentication and authorization are related but separate
things. A client may be authenticated but not authorized to consume
messages from or send messages to specific topics. This is a fundamental
concept of role-based access control.
The CONNECT packet also contains all the details about the LWT message
(i.e. payload, properties, & topic). However, authorization for actually
sending the will message is not performed at this point. Authorization is
only performed when the LWT message is actually sent (e.g. when the
client's connection fails). All sending operations are authorized at the
time they occur whether that's via the normal PUBLISH packet from the
client or for a LWT message sent on the client's behalf by the broker.
Authorization is important here because the destination topic for the LWT
message is arbitrary. If authorization was not performed then it would be
simple for a client to send a message to a topic which it would not
otherwise have authorization.
If authorization is failing when the broker attempts to send the client's
LWT message and you're circumventing this so that the message is actually
sent then it seems you may be undermining the security of your environment.
You may inadvertently allow a clever, nefarious actor to send a message to
a topic to which they are not authorized.
How are you currently re-authenticating your clients when their JWTs expire?
Justin
On Fri, Oct 24, 2025 at 3:19 PM Shields, Paul <pa...@hpe.com.invalid>
wrote:
> Hi,
>
> Why are MQTT last will messages authenticated with Artemis before being
> sent? From my understanding of the MQTT last will feature is that
> authentication is a separate step handled during the client's initial
> connection. The LWT itself is a message prepared by a client and stored by
> the broker to be published only if the client disconnects unexpectedly. The
> security of the LWT message is therefore dependent on the security of the
> initial client connection, which must be established through methods like
> username/password or TLS and not when the LWT is delivered/published.
>
> We have written a custom securityManager plugin that uses Json Web Tokens
> as passwords for connecting MQTT clients. We use MQTT for server
> availability which has a state of either up or down. When a server client
> connects with the MQTT Broker it is authenticated with the broker using a
> JWT, registers a last will, and then publishes a server up message on a
> MQTT topic. We have other MQTT clients that also connect with the Artemis
> broker using JWTs for auth and then subscribe to server state topics. The
> MQTT last will is used to publish server down messages when the long
> running server dies for some reason. Some of the servers publish
> “re-announce” messages as they complete certain steps of processing that
> they publish on a different topic using the initial client connection.
> This was initially developed using Artemis 2.28.0 and later used in Artemis
> 2.34.0 and 2.40.0. Our securityManager plugin, JwtJassSecurityManager,
> implements the ActiveMQSecurityManager5 interface. During our initial
> development of the securityManager plugin we saw MQTT last will messages
> failing authentication and put a workaround in place to check that if there
> is a subject and a principal with the same user name (each MQTT client has
> a unique user name) which means that this is the same session that has
> previously been authenticated, we will validate the JWT, but ignore the
> expiration time. We are now upgrading to Artemis 2.43.0 and would like to
> remove the workaround. Our workaround for last will authentication breaks
> down when the keys are rotated. It is about to become a real problem as we
> are going to rotate the JWT key on a more frequent schedule. Here are the
> errors produced:
>
> 2025-10-14 15:06:51,858 INFO
> [com.hpe.hpc.activemq.JwtJaasSecurityManager] validation failed: username:
> x3000c0s9b0n0, details: auth is invalid
> InvalidJwtException: JWT processing failed. Additional details: [[17]
> Unable to process JOSE object (cause:
> org.jose4j.lang.UnresolvableKeyException: Unable to find a suitable
> verification key for JWS w/ header {"alg":"RS256","kid”:”previous
> KEY-A","typ":"JWT"} from JWKs [org.jose4j.jwk.RsaJsonWebKey{kty=RSA,
> kid=KEY-A, alg=RS256, n=CENSORED, e=CENSORED},
> org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-B, alg=RS256, n=CENSORED,
> e=CENSORED}, org.jose4j.jwk.RsaJsonWebKey{kty=RSA, kid=KEY-C, alg=RS256,
> n=CENSORED, e=CENSORED}]): JsonWebSignature{"alg":"RS256","kid”:”previous
> KEY-A","typ":”JWT”CENSORED
> ]
> 2025-10-14 15:06:51,861 INFO
> [com.hpe.hpc.activemq.JwtJaasSecurityManager] Invalid authentication for
> user: x3000c0s9b0n0, subject: Subject:
> Principal: pzxQ2XQN
> Principal: admin
> Principal: x3000c0s9b0n0
>
> 2025-10-14 15:06:51,863 WARN [org.apache.activemq.artemis.core.server]
> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
> certificate subject DN: unavailable
> 2025-10-14 15:06:51,863 ERROR
> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error
> processing control packet:
> MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH,
> isDup=false, qosLevel=AT_MOST_ONCE, isRetain=true, remainingLength=67],
> variableHeader=MqttPublishVariableHeader[topicName=trusted/x3000c0s9b0n0/dvs/server/state,
> packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 27, cap: 27/27,
> unwrapped: PooledUnsafeDirectByteBuf(ridx: 69, widx: 69, cap: 80))]
> org.apache.activemq.artemis.api.core.ActiveMQSecurityException: AMQ229031:
> Unable to validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0;
> SSL certificate subject DN: unavailable
> at
> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticationFailed(SecurityStoreImpl.java:449)
> at
> org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:341)
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:527)
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2365)
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1934)
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendToQueue(MQTTPublishManager.java:242)
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:322)
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(MQTTProtocolHandler.java:164)
> at
> org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:32)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
> 2025-10-14 15:06:51,866 WARN [org.apache.activemq.artemis.core.server]
> AMQ222216: Security problem while authenticating: AMQ229031: Unable to
> validate user from 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL
> certificate subject DN: unavailable
> 2025-10-14 15:06:51,866 ERROR
> [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834007: Authorization
> failure sending will message: AMQ229031: Unable to validate user from
> 127.0.0.6:49859. Username: x3000c0s9b0n0; SSL certificate subject DN:
> unavailable
>
> Regards,
> Paul Shields
>