SharedFileLocker.keepAlive() can block indefinitely on hard-mounted NFS (distinct from AMQ-5568)

[Axel Bommersbach via dev <de...@activemq.apache.org>]

Hi all,

We've been investigating split-brain behavior with ActiveMQ Classic 6.2.7 using KahaDB's shared-file-lock mechanism over a hard-mounted NFSv4 share (AWS FSx), and found what looks like a separate issue from AMQ-5568.

Summary: When the NFS server becomes unreachable while hard-mounted, LockFile.hasBeenModified() (called from SharedFileLocker.keepAlive()) can block indefinitely in the underlying native stat() syscall, since a hard NFS mount deliberately never returns an error — it waits for the server to come back. Because keepAlive() itself never returns, the broker never reaches the stopOnError/stopBroker() path that's supposed to detect lock loss. The broker keeps running as a "zombie master": its transport connectors stay open and keep accepting connections, while the standby broker (once the NFSv4 client lease/grace period elapses) can acquire the lock and become a second, independent master.

How this differs from AMQ-5568: That issue was about deleting the lock file during shutdown potentially bringing down the new master too, and was fixed by making deleteOnUnlock always false. That fix doesn't touch this issue, since in our case keepAlive() hangs before any shutdown/delete path is ever reached.

Reproduction (locally, Docker Compose, two brokers sharing one NFSv4-mounted KahaDB store):

  *   Simulate NFS server unreachability for the current master only (iptables DROP on ports 2049/111 from that container's network namespace, not the NFS server itself)
  *   Thread dump (SIGQUIT, since jstack isn't in the JRE-only image) shows multiple "ActiveMQ Lock KeepAlive Timer" threads stuck in:

at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.hasBooleanAttributes
at java.io.File.exists
at org.apache.activemq.util.LockFile.hasBeenModified(LockFile.java:167)
at org.apache.activemq.util.LockFile.keepAlive(LockFile.java:183)
at org.apache.activemq.store.SharedFileLocker.keepAlive(SharedFileLocker.java:97)
at org.apache.activemq.broker.LockableServiceSupport.keepLockAlive(LockableServiceSupport.java:137)

  *   Independently confirmed: timeout 5 stat <kahadb-dir> from the host returns exit code 124 (timeout), not an error, for the whole duration of the outage.

We don't currently have a JIRA account - wanted to check here first whether this is a known/expected limitation, whether it's worth a new ticket, or whether we're missing an existing configuration option that addresses it, before filing anything.

Thanks,

Axel

AW: SharedFileLocker.keepAlive() can block indefinitely on hard-mounted NFS (distinct from AMQ-5568)

[Axel Bommersbach via dev <de...@activemq.apache.org>]

Hi Hailey,

Thanks for confirming this reproduces independently of AMQ-5568, and for the pointer on the keepAlive-heartbeat idea.

      > My suggestion for upstream fix would be to have the keepAlive emit some sort of heartbeat and treat timeouts on that heartbeat as a lock loss.
      > Also, something to note that lockKeepAlivePeriod defaults to 0, meaning a default shared-store config will still be susceptible to this issue even
      > after the fix, unless that default changes too.

Good catch - that default would need to change alongside any fix, or the heartbeat never fires for anyone running defaults.

On our side, we ended up building a stop-gap while waiting on an upstream fix: a custom Locker that wraps lock()/keepAlive()/unlock() in a timed executor and calls Runtime.halt() on timeout, since a hard-mounted NFS hang leaves the affected thread in D-state and unresponsive to normal shutdown or even SIGKILL. It's not something we'd propose as *the* fix - more a symptom of how deep this problem sits, and evidence that a timeout-based approach at the locker level is workable in practice. It's only been tested in our own reproduction setup so far, not reviewed by a colleague yet.

Happy to push it to a public branch if that's useful as a reference. Would a JIRA ticket be the right place to attach something like that, or is this still better kept at the discussion stage for now?

Best,
Axel

Re: SharedFileLocker.keepAlive() can block indefinitely on hard-mounted NFS (distinct from AMQ-5568)

["Bentz, Hailey" <ha...@amazon.com>]

Hey Axel,

I've been able to reproduce this issue as far back as 5.18.6 using your steps, so I think this is long-standing, and different than AMQ-5568.

This behaviour appears to be independent of any specific AWS FSx behaviour and should happen with any hard-mounted NFS share.

My suggestion for upstream fix would be to have the keepAlive emit some sort of heartbeat and treat timeouts on that heartbeat as a lock loss. Also, something to note that lockKeepAlivePeriod defaults to 0, meaning a default shared-store config will still be susceptible to this issue even after the fix, unless that default changes too.

Best,

Hailey

On 2026-08-12, 02:55, "Axel Bommersbach via dev" <dev@activemq.apache.org <ma...@activemq.apache.org>> wrote:


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.






AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.






Hi all,


We've been investigating split-brain behavior with ActiveMQ Classic 6.2.7 using KahaDB's shared-file-lock mechanism over a hard-mounted NFSv4 share (AWS FSx), and found what looks like a separate issue from AMQ-5568.


Summary: When the NFS server becomes unreachable while hard-mounted, LockFile.hasBeenModified() (called from SharedFileLocker.keepAlive()) can block indefinitely in the underlying native stat() syscall, since a hard NFS mount deliberately never returns an error — it waits for the server to come back. Because keepAlive() itself never returns, the broker never reaches the stopOnError/stopBroker() path that's supposed to detect lock loss. The broker keeps running as a "zombie master": its transport connectors stay open and keep accepting connections, while the standby broker (once the NFSv4 client lease/grace period elapses) can acquire the lock and become a second, independent master.


How this differs from AMQ-5568: That issue was about deleting the lock file during shutdown potentially bringing down the new master too, and was fixed by making deleteOnUnlock always false. That fix doesn't touch this issue, since in our case keepAlive() hangs before any shutdown/delete path is ever reached.


Reproduction (locally, Docker Compose, two brokers sharing one NFSv4-mounted KahaDB store):


* Simulate NFS server unreachability for the current master only (iptables DROP on ports 2049/111 from that container's network namespace, not the NFS server itself)
* Thread dump (SIGQUIT, since jstack isn't in the JRE-only image) shows multiple "ActiveMQ Lock KeepAlive Timer" threads stuck in:


at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.hasBooleanAttributes
at java.io.File.exists
at org.apache.activemq.util.LockFile.hasBeenModified(LockFile.java:167)
at org.apache.activemq.util.LockFile.keepAlive(LockFile.java:183)
at org.apache.activemq.store.SharedFileLocker.keepAlive(SharedFileLocker.java:97)
at org.apache.activemq.broker.LockableServiceSupport.keepLockAlive(LockableServiceSupport.java:137)


* Independently confirmed: timeout 5 stat <kahadb-dir> from the host returns exit code 124 (timeout), not an error, for the whole duration of the outage.


We don't currently have a JIRA account - wanted to check here first whether this is a known/expected limitation, whether it's worth a new ticket, or whether we're missing an existing configuration option that addresses it, before filing anything.


Thanks,


Axel