You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Alex Rudyy (JIRA)" <ji...@apache.org> on 2018/03/27 09:41:00 UTC
[jira] [Updated] (QPID-8140) [Broker-J][BDB HA] Removal of non
existing group member can end up in broker crash due to uncaught
MemberNotFoundException
[ https://issues.apache.org/jira/browse/QPID-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Rudyy updated QPID-8140:
-----------------------------
Description:
When two concurrent requests are made to delete a group member using REST API for remote replication node, one can successfully remove the node, whilst other can end-up in unhandled {{MemberNotFoundException}} which crashes the broker.
{code}
http://localhost:8080/api/latest/remotereplicationnode/node2/node1
{code}
For failed requests an exception like the one below is reported
{noformat}
ERROR [VirtualHostNode-node2-Config] o.a.q.s.u.ServerScopedRuntimeException Exception on node removal from group
com.sleepycat.je.rep.MemberNotFoundException: (JE 7.4.5) Node: node1 is not currently a member of the group: BDBHAVirtualHostNodeRestTest.testDeleteMasterNode, it has been removed.
at com.sleepycat.je.rep.util.ReplicationGroupAdmin.checkMember(ReplicationGroupAdmin.java:576)
at com.sleepycat.je.rep.util.ReplicationGroupAdmin.removeMember(ReplicationGroupAdmin.java:314)
at org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.removeNodeFromGroup(ReplicatedEnvironmentFacade.java:1210)
at org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHARemoteReplicationNodeImpl.onDelete(BDBHARemoteReplicationNodeImpl.java:139)
at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2759)
at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:822)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:664)
at org.apache.qpid.server.model.AbstractConfiguredObject$ChainedSettableFuture.set(AbstractConfiguredObject.java:2710)
at org.apache.qpid.server.model.AbstractConfiguredObject$21$1.onSuccess(AbstractConfiguredObject.java:2765)
at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
at org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2884)
at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2760)
at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:415)
at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:411)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1.run(TaskExecutorImpl.java:410)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
When two concurrent requests are made to delete a group member using REST API for remote replication node, one can successfully remove the node, whilst other can end-up in unhandled {{MemberNotFoundException}} which crashes the broker.
{code}
http://localhost:8080/api/latest/remotereplicationnode/node2/node1
{code}
For failed requests an exception like the one below is reported
{noformat}
ERROR [VirtualHostNode-node2-Config] o.a.q.s.u.ServerScopedRuntimeException Exception on node removal from group com.sleepycat.je.rep.MemberNotFoundException: (JE 7.4.5) Node: node1 is not currently a member of the group: BDBHAVirtualHostNodeRestTest.testDeleteMasterNode, it has been removed. at com.sleepycat.je.rep.util.ReplicationGroupAdmin.checkMember(ReplicationGroupAdmin.java:576) at com.sleepycat.je.rep.util.ReplicationGroupAdmin.removeMember(ReplicationGroupAdmin.java:314) at org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.removeNodeFromGroup(ReplicatedEnvironmentFacade.java:1210) at org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHARemoteReplicationNodeImpl.onDelete(BDBHARemoteReplicationNodeImpl.java:139) at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2759) at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894) at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237) at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400) at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:822) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:664) at org.apache.qpid.server.model.AbstractConfiguredObject$ChainedSettableFuture.set(AbstractConfiguredObject.java:2710) at org.apache.qpid.server.model.AbstractConfiguredObject$21$1.onSuccess(AbstractConfiguredObject.java:2765) at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894) at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237) at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400) at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911) at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101) at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209) at org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2884) at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2760) at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894) at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237) at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:415) at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:411) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1.run(TaskExecutorImpl.java:410) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464) at java.lang.Thread.run(Thread.java:748)
{noformat}
> [Broker-J][BDB HA] Removal of non existing group member can end up in broker crash due to uncaught MemberNotFoundException
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-8140
> URL: https://issues.apache.org/jira/browse/QPID-8140
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
> Affects Versions: qpid-java-broker-7.0.2, qpid-java-broker-7.0.0, qpid-java-broker-7.0.1
> Reporter: Alex Rudyy
> Priority: Major
>
> When two concurrent requests are made to delete a group member using REST API for remote replication node, one can successfully remove the node, whilst other can end-up in unhandled {{MemberNotFoundException}} which crashes the broker.
> {code}
> http://localhost:8080/api/latest/remotereplicationnode/node2/node1
> {code}
> For failed requests an exception like the one below is reported
> {noformat}
> ERROR [VirtualHostNode-node2-Config] o.a.q.s.u.ServerScopedRuntimeException Exception on node removal from group
> com.sleepycat.je.rep.MemberNotFoundException: (JE 7.4.5) Node: node1 is not currently a member of the group: BDBHAVirtualHostNodeRestTest.testDeleteMasterNode, it has been removed.
> at com.sleepycat.je.rep.util.ReplicationGroupAdmin.checkMember(ReplicationGroupAdmin.java:576)
> at com.sleepycat.je.rep.util.ReplicationGroupAdmin.removeMember(ReplicationGroupAdmin.java:314)
> at org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.removeNodeFromGroup(ReplicatedEnvironmentFacade.java:1210)
> at org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHARemoteReplicationNodeImpl.onDelete(BDBHARemoteReplicationNodeImpl.java:139)
> at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2759)
> at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
> at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
> at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
> at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
> at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
> at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
> at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:822)
> at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:664)
> at org.apache.qpid.server.model.AbstractConfiguredObject$ChainedSettableFuture.set(AbstractConfiguredObject.java:2710)
> at org.apache.qpid.server.model.AbstractConfiguredObject$21$1.onSuccess(AbstractConfiguredObject.java:2765)
> at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
> at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
> at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
> at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
> at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
> at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
> at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
> at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
> at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
> at org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2884)
> at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2760)
> at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
> at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
> at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
> at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:415)
> at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:411)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1.run(TaskExecutorImpl.java:410)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org