You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by nexus <an...@mastersoft.lv> on 2020/03/21 14:03:50 UTC
Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Hello,
Encountering following issue with network of brokers on Windows 10 / 64-bit
/ ActiveMQ v5.15.12
Problem
Cannot re-establish duplex network connection on remote broker with
following error:
Remote broker is configured to delete inactive destinations
<https://activemq.apache.org/delete-inactive-destinations.html> and uses
mKahaDB persistence adapters for different destinations ( Automatic Per
Destination Persistence Adapter <https://activemq.apache.org/kahadb> ).
Same setup, but single kahaDB persistence adapter on remote broker is not
causing the issue.
See attached files for detailed configuration and logs:
* local-activemq.xml
<http://activemq.2283324.n4.nabble.com/file/t379838/local-activemq.xml>
* good-remote-activemq.xml
<http://activemq.2283324.n4.nabble.com/file/t379838/good-remote-activemq.xml>
* good-remote-wrapper.log
<http://activemq.2283324.n4.nabble.com/file/t379838/good-remote-wrapper.log>
* bad-remote-activemq.xml
<http://activemq.2283324.n4.nabble.com/file/t379838/bad-remote-activemq.xml>
* bad-remote-wrapper.log
<http://activemq.2283324.n4.nabble.com/file/t379838/bad-remote-wrapper.log>
Use case
Simulate network connection loss and then re-astablish duplex communication
after remote broker destinations were purged due to inactivity:
1. start remote broker
2. start local broker
3. destination queue created automatically on remote (active consumer from
local broker is also shown correctly in web-console)
4. stop local broker
5. wait for a while until destination is deleted on remote due to inactivity
6. start local broker again
Results
GOOD: single kahaDB persistence adapter on remote broker
BAD: multiple kahaDb persistence adapters on remote broker with
perDestination="true"
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Posted by Tim Bain <tb...@alumni.duke.edu>.
I had forgotten you'd already submitted one. Thank you for doing that.
Tim
On Sat, Mar 28, 2020, 12:13 AM nexus <an...@mastersoft.lv> wrote:
> I have already opened an issue AMQ-7453 with detailed description, logs and
> examples, please, take a look:
> https://issues.apache.org/jira/browse/AMQ-7453
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Posted by nexus <an...@mastersoft.lv>.
I have already opened an issue AMQ-7453 with detailed description, logs and
examples, please, take a look:
https://issues.apache.org/jira/browse/AMQ-7453
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Posted by Tim Bain <tb...@alumni.duke.edu>.
Thanks for testing that; it will help make sure that whoever works on a fix
focuses quickly on the right part of the code. Would you be able to submit
a bug in JIRA to capture the problem as you observed it? If you're able to
attach any config files that would make it easier to quickly reproduce the
problem, that would be helpful as well.
Thanks for working through this preliminary testing and helping better
characterize the problem.
Tim
On Fri, Mar 27, 2020, 12:55 AM nexus <an...@mastersoft.lv> wrote:
> > if you cycle the remote broker while the local one is down, do you get
> the
> same behavior?
>
> I've performed the test, there's no error after clean restart of the remote
> broker, but the problem repeats whenever inactive destinations are purged
> once again. So, I suppose, your assumption is right "it is purely something
> in-memory on the remote broker".
>
> bad-activemq-restart.log
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/bad-activemq-restart.log>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Posted by nexus <an...@mastersoft.lv>.
> if you cycle the remote broker while the local one is down, do you get the
same behavior?
I've performed the test, there's no error after clean restart of the remote
broker, but the problem repeats whenever inactive destinations are purged
once again. So, I suppose, your assumption is right "it is purely something
in-memory on the remote broker".
bad-activemq-restart.log
<http://activemq.2283324.n4.nabble.com/file/t379838/bad-activemq-restart.log>
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Posted by Tim Bain <tb...@alumni.duke.edu>.
What I meant was, if you cycle the remote broker while the local one is
down, do you get the same behavior? I'm trying to characterize whether the
problem is purely something in-memory on the remote broker, or whether the
problem is also in the content written to disk (and therefore the problem
persists even if you cycle the remote broker).
Tim
On Thu, Mar 26, 2020, 1:38 AM nexus <an...@mastersoft.lv> wrote:
> > can you please submit a bug report for it in JIRA?
>
> Done:
> https://issues.apache.org/jira/projects/AMQ/issues/AMQ-7453
>
> > whether the problem occurs if you cycle the remote broker between steps 5
> > and 6
>
> Steps 5. - 6. can be repeated multiple times -- problem persists.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Posted by nexus <an...@mastersoft.lv>.
> can you please submit a bug report for it in JIRA?
Done:
https://issues.apache.org/jira/projects/AMQ/issues/AMQ-7453
> whether the problem occurs if you cycle the remote broker between steps 5
> and 6
Steps 5. - 6. can be repeated multiple times -- problem persists.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Posted by Tim Bain <tb...@alumni.duke.edu>.
This definitely sounds like a bug in the mKahaDB code, so can you please
submit a bug report for it in JIRA?
I'd be curious to know whether the problem occurs if you cycle the remote
broker between steps 5 and 6. If not, it's likely something in-memory that
represents an inconsistency between the filesystem and what's in memory for
the mKahaDB database within the remote broker.
Tim
On Sun, Mar 22, 2020, 4:03 AM nexus <an...@mastersoft.lv> wrote:
> > Can you please provide a listing (ls -l) of the KahaDB data directory?
> See details below aligned with use case.
> > Also, can you please configure mKahaDB to point to a different (empty)
> > directory rather than the same one that the normal KahaDB points to, and
> > try again? I've never used mKahaDB so I don't have direct experience, but
> > I wouldn't expect the broker to be happy if you switch between KahaDB and
> > mKahaDB using the same data directory (and the same files within it, more
> > importantly). Unless your process for switching the KahaDB type already
> > involved deleting the files in the data directory?
> There was no switch from single to multi kahadb. Problem is reproduced with
> clean installation of apache-activemq-5.15.12-bin.zip
> <
> http://www.apache.org/dyn/closer.cgi?filename=/activemq/5.15.12/apache-activemq-5.15.12-bin.zip&action=download>
>
> on 2 PCs and using configuration for local and remote brokers I a have
> provided in the first post.
> Some observations
> The main difference I see in logs, that in bad situtation (mKahaDB +
> perDestination="true") following messages appear after inactive
> destinations
> deleted:
> Since this moment local broker cannot establish duplex connection any more,
> and it doesn't matter which destinations has been purged -- with the same
> name (App.Data) or any other. Also it doesn't matter whether local broker
> already had any successful communication with the remote. As soon as these
> messages appear, broker cannot "create responder end of duplex network
> bridge" because of "PageFile is not loaded".
> When I try to do the same with single kahadb instance, these messages do
> not
> appear and no such problem.
> Use case with logs and kahadb listing
> Attached logs
> activemq.log
> <http://activemq.2283324.n4.nabble.com/file/t379838/activemq.log>
> wrapper.log <
> http://activemq.2283324.n4.nabble.com/file/t379838/wrapper.log>
> 1. Remote broker started
> kahadb
> 2. Local broker started
> Duplex network connection established.
> Statically included queue "App.Data" created on remote.
> Active consumer is shown in web-console.
> wrapper.log
> kahadb
> 3. Local broker stopped and remote deletes inactive destinations after a
> while
> wrapper.log
> kahadb
> 4. Local broker started again
> wrapper.log
> kahadb
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Posted by nexus <an...@mastersoft.lv>.
> Can you please provide a listing (ls -l) of the KahaDB data directory?
See details below aligned with use case.
> Also, can you please configure mKahaDB to point to a different (empty)
> directory rather than the same one that the normal KahaDB points to, and
> try again? I've never used mKahaDB so I don't have direct experience, but
> I wouldn't expect the broker to be happy if you switch between KahaDB and
> mKahaDB using the same data directory (and the same files within it, more
> importantly). Unless your process for switching the KahaDB type already
> involved deleting the files in the data directory?
There was no switch from single to multi kahadb. Problem is reproduced with
clean installation of apache-activemq-5.15.12-bin.zip
<http://www.apache.org/dyn/closer.cgi?filename=/activemq/5.15.12/apache-activemq-5.15.12-bin.zip&action=download>
on 2 PCs and using configuration for local and remote brokers I a have
provided in the first post.
Some observations
The main difference I see in logs, that in bad situtation (mKahaDB +
perDestination="true") following messages appear after inactive destinations
deleted:
Since this moment local broker cannot establish duplex connection any more,
and it doesn't matter which destinations has been purged -- with the same
name (App.Data) or any other. Also it doesn't matter whether local broker
already had any successful communication with the remote. As soon as these
messages appear, broker cannot "create responder end of duplex network
bridge" because of "PageFile is not loaded".
When I try to do the same with single kahadb instance, these messages do not
appear and no such problem.
Use case with logs and kahadb listing
Attached logs
activemq.log
<http://activemq.2283324.n4.nabble.com/file/t379838/activemq.log>
wrapper.log <http://activemq.2283324.n4.nabble.com/file/t379838/wrapper.log>
1. Remote broker started
kahadb
2. Local broker started
Duplex network connection established.
Statically included queue "App.Data" created on remote.
Active consumer is shown in web-console.
wrapper.log
kahadb
3. Local broker stopped and remote deletes inactive destinations after a
while
wrapper.log
kahadb
4. Local broker started again
wrapper.log
kahadb
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Duplex networkConnector error with mKahaDB after inactive
destinations deleted
Posted by Tim Bain <tb...@alumni.duke.edu>.
Can you please provide a listing (ls -l) of the KahaDB data directory?
Also, can you please configure mKahaDB to point to a different (empty)
directory rather than the same one that the normal KahaDB points to, and
try again? I've never used mKahaDB so I don't have direct experience, but I
wouldn't expect the broker to be happy if you switch between KahaDB and
mKahaDB using the same data directory (and the same files within it, more
importantly). Unless your process for switching the KahaDB type already
involved deleting the files in the data directory?
Tim
Tim
On Sat, Mar 21, 2020, 8:03 AM nexus <an...@mastersoft.lv> wrote:
> Hello,
>
> Encountering following issue with network of brokers on Windows 10 / 64-bit
> / ActiveMQ v5.15.12
>
> Problem
>
>
> Cannot re-establish duplex network connection on remote broker with
> following error:
>
>
>
> Remote broker is configured to delete inactive destinations
> <https://activemq.apache.org/delete-inactive-destinations.html> and uses
> mKahaDB persistence adapters for different destinations ( Automatic Per
> Destination Persistence Adapter <https://activemq.apache.org/kahadb> ).
>
> Same setup, but single kahaDB persistence adapter on remote broker is not
> causing the issue.
>
> See attached files for detailed configuration and logs:
>
> * local-activemq.xml
> <http://activemq.2283324.n4.nabble.com/file/t379838/local-activemq.xml>
> * good-remote-activemq.xml
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/good-remote-activemq.xml>
>
> * good-remote-wrapper.log
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/good-remote-wrapper.log>
>
> * bad-remote-activemq.xml
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/bad-remote-activemq.xml>
>
> * bad-remote-wrapper.log
> <http://activemq.2283324.n4.nabble.com/file/t379838/bad-remote-wrapper.log>
>
>
> Use case
>
>
> Simulate network connection loss and then re-astablish duplex communication
> after remote broker destinations were purged due to inactivity:
>
> 1. start remote broker
> 2. start local broker
> 3. destination queue created automatically on remote (active consumer from
> local broker is also shown correctly in web-console)
> 4. stop local broker
> 5. wait for a while until destination is deleted on remote due to
> inactivity
> 6. start local broker again
>
> Results
>
>
> GOOD: single kahaDB persistence adapter on remote broker
>
>
>
>
> BAD: multiple kahaDb persistence adapters on remote broker with
> perDestination="true"
>
>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>