You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Keith Wall (JIRA)" <ji...@apache.org> on 2016/11/11 09:27:59 UTC
[jira] [Updated] (QPID-7025) [JavaBroker, BDB HA] Application state
is not set on JE Node when BDB HA VHN role changes from MASTER to DETACHED
and back to MASTER
[ https://issues.apache.org/jira/browse/QPID-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Wall updated QPID-7025:
-----------------------------
Description:
Whilst investigating a failure of test MultiNodeTest#testInFlightTransactionsWhilstMajorityIsLost I noticed that Replica node was logging null application state which was received as part of invocation of DbPing(r...)#getNodeState().
Test failure looks like this:
{noformat}
Error Message
Unexpected role at 10003
Stacktrace
java.lang.AssertionError: Unexpected role at 10003
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.apache.qpid.server.store.berkeleydb.replication.GroupCreator.awaitNodeToAttainAttributeValue(GroupCreator.java:472)
at org.apache.qpid.server.store.berkeleydb.replication.GroupCreator.awaitNodeToAttainRole(GroupCreator.java:448)
at org.apache.qpid.server.store.berkeleydb.replication.GroupCreator.awaitNodeToAttainRole(GroupCreator.java:443)
at org.apache.qpid.server.store.berkeleydb.replication.MultiNodeTest.testInFlightTransactionsWhilstMajorityIsLost(MultiNodeTest.java:611)
{noformat}
The log is below:
{noformat}
2016-01-27 17:05:07,569 BROKER-19 DEBUG [Group-Change-Learner:test:nodetestInFlightTransactionsWhilstMajorityIsLost10007] o.a.q.s.v.b.BDBHAVirtualHostNodeImpl Application state returned by JE was 'null' so skipping permitted node handling: Current state of node: nodetestInFlightTransactionsWhilstMajorityIsLost10001 from group: test
Current state: MASTER
Current master: nodetestInFlightTransactionsWhilstMajorityIsLost10001
Current JE version: 5.0.104
Current log version: 8
Current transaction end (abort or commit) VLSN: 344
Current master transaction end (abort or commit) VLSN: 0
Current active feeders on node: 1
Current system load average: 58.23
{noformat}
The logs in the test log file indicate that after transition from MASTER into DETACHED (after lost of majority) and back to MASTER after restoring full cluster, the application state was not set on environment after flipping back to MASTER.
The information about permitted nodes is not communicated to replicas.
It seems that the issue will not affect an existing fully formed cluster as replica nodes will continue using locally stored permitted nodes. When permitted nodes are updated on a Master, the updated permitted nodes will be set as application state and broadcasted to the nodes in the cluster.
The issue might only affect joining of a new node to the group. For example, in the following scenario a new node will not be able to join the group:
Lets assume we want to build 3 nodes cluster and created 1st and 2d nodes. Lets assume that before creation of 3rd node, MASTER node role changed from MASTER to DETACHED and back to MASTER. In result of such transitions , application state is not set on environment and MASTER is not broadcasting permitted nodes. If user will try to add 3rd node, that will fail when MASTER node is specified as as helper node because permitted nodes will not be set on the new replica node and configured object opening will result in validation check failure
was:
Whilst investigating a failure of test MultiNodeTest#testInFlightTransactionsWhilstMajorityIsLost I noticed that Replica node was logging null application state which was received as part of invocation of DbPing(r...)#getNodeState().
The log is below:
{noformat}
2016-01-27 17:05:07,569 BROKER-19 DEBUG [Group-Change-Learner:test:nodetestInFlightTransactionsWhilstMajorityIsLost10007] o.a.q.s.v.b.BDBHAVirtualHostNodeImpl Application state returned by JE was 'null' so skipping permitted node handling: Current state of node: nodetestInFlightTransactionsWhilstMajorityIsLost10001 from group: test
Current state: MASTER
Current master: nodetestInFlightTransactionsWhilstMajorityIsLost10001
Current JE version: 5.0.104
Current log version: 8
Current transaction end (abort or commit) VLSN: 344
Current master transaction end (abort or commit) VLSN: 0
Current active feeders on node: 1
Current system load average: 58.23
{noformat}
The logs in the test log file indicate that after transition from MASTER into DETACHED (after lost of majority) and back to MASTER after restoring full cluster, the application state was not set on environment after flipping back to MASTER.
The information about permitted nodes is not communicated to replicas.
It seems that the issue will not affect an existing fully formed cluster as replica nodes will continue using locally stored permitted nodes. When permitted nodes are updated on a Master, the updated permitted nodes will be set as application state and broadcasted to the nodes in the cluster.
The issue might only affect joining of a new node to the group. For example, in the following scenario a new node will not be able to join the group:
Lets assume we want to build 3 nodes cluster and created 1st and 2d nodes. Lets assume that before creation of 3rd node, MASTER node role changed from MASTER to DETACHED and back to MASTER. In result of such transitions , application state is not set on environment and MASTER is not broadcasting permitted nodes. If user will try to add 3rd node, that will fail when MASTER node is specified as as helper node because permitted nodes will not be set on the new replica node and configured object opening will result in validation check failure
> [JavaBroker, BDB HA] Application state is not set on JE Node when BDB HA VHN role changes from MASTER to DETACHED and back to MASTER
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-7025
> URL: https://issues.apache.org/jira/browse/QPID-7025
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: qpid-java-6.0.1, qpid-java-6.1
> Reporter: Alex Rudyy
> Attachments: TEST-org.apache.qpid.server.store.berkeleydb.replication.MultiNodeTest.testInFlightTransactionsWhilstMajorityIsLost.txt
>
>
> Whilst investigating a failure of test MultiNodeTest#testInFlightTransactionsWhilstMajorityIsLost I noticed that Replica node was logging null application state which was received as part of invocation of DbPing(r...)#getNodeState().
> Test failure looks like this:
> {noformat}
> Error Message
> Unexpected role at 10003
> Stacktrace
> java.lang.AssertionError: Unexpected role at 10003
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.apache.qpid.server.store.berkeleydb.replication.GroupCreator.awaitNodeToAttainAttributeValue(GroupCreator.java:472)
> at org.apache.qpid.server.store.berkeleydb.replication.GroupCreator.awaitNodeToAttainRole(GroupCreator.java:448)
> at org.apache.qpid.server.store.berkeleydb.replication.GroupCreator.awaitNodeToAttainRole(GroupCreator.java:443)
> at org.apache.qpid.server.store.berkeleydb.replication.MultiNodeTest.testInFlightTransactionsWhilstMajorityIsLost(MultiNodeTest.java:611)
> {noformat}
> The log is below:
> {noformat}
> 2016-01-27 17:05:07,569 BROKER-19 DEBUG [Group-Change-Learner:test:nodetestInFlightTransactionsWhilstMajorityIsLost10007] o.a.q.s.v.b.BDBHAVirtualHostNodeImpl Application state returned by JE was 'null' so skipping permitted node handling: Current state of node: nodetestInFlightTransactionsWhilstMajorityIsLost10001 from group: test
> Current state: MASTER
> Current master: nodetestInFlightTransactionsWhilstMajorityIsLost10001
> Current JE version: 5.0.104
> Current log version: 8
> Current transaction end (abort or commit) VLSN: 344
> Current master transaction end (abort or commit) VLSN: 0
> Current active feeders on node: 1
> Current system load average: 58.23
> {noformat}
> The logs in the test log file indicate that after transition from MASTER into DETACHED (after lost of majority) and back to MASTER after restoring full cluster, the application state was not set on environment after flipping back to MASTER.
> The information about permitted nodes is not communicated to replicas.
> It seems that the issue will not affect an existing fully formed cluster as replica nodes will continue using locally stored permitted nodes. When permitted nodes are updated on a Master, the updated permitted nodes will be set as application state and broadcasted to the nodes in the cluster.
> The issue might only affect joining of a new node to the group. For example, in the following scenario a new node will not be able to join the group:
> Lets assume we want to build 3 nodes cluster and created 1st and 2d nodes. Lets assume that before creation of 3rd node, MASTER node role changed from MASTER to DETACHED and back to MASTER. In result of such transitions , application state is not set on environment and MASTER is not broadcasting permitted nodes. If user will try to add 3rd node, that will fail when MASTER node is specified as as helper node because permitted nodes will not be set on the new replica node and configured object opening will result in validation check failure
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org