Repetitive bridge queue delete

[Scott Nicol <sn...@tessellationsoftware.com>]

I'm working on a project using qpid, and we have been having stability
problems with the brokers. We are using version 0.32 C++ brokers and Java
clients. Two brokers in a federation sharing a bunch of exchanges.
Initially we used dynamic routes, but switched to static routes to see if
that affected stability.

I've been stress-testing the broker trying to reproduce reported problems,
and found a few. I still don't have easily reproducible cases, for example
this one took four hours of hammering the brokers.

A summary of the problem:

The FAST exchange goes down. The first line on the log shows its queue is
deleted. After this point, no traffic flows over the FAST exchange, and the
route cannot be reestablished without shutting down the brokers.

However, 1 second later, and repeated every 2 seconds for 4 hours, the TAP
exchange is deleted and presumably brought back up again, as messages
continue to flow across this exchange.

>From the qpidd log:

Tue Apr 14 20:30:29 2015 org.apache.qpid.broker:queueDelete user=
qName=qpid.bridge_queue_qpid.tcp:dev3:10089!FAST!FAST!securities.#_4f37e743-b73c-4cc1-80bb-7348a254d36d
rhost=
Tue Apr 14 20:30:30 2015 org.apache.qpid.broker:queueDelete user=
qName=qpid.bridge_queue_qpid.tcp:dev3:10089!TAP!TAP!securities.#_4f37e743-b73c-4cc1-80bb-7348a254d36d
rhost=
Tue Apr 14 20:30:32 2015 org.apache.qpid.broker:queueDelete user=
qName=qpid.bridge_queue_qpid.tcp:dev3:10089!TAP!TAP!securities.#_4f37e743-b73c-4cc1-80bb-7348a254d36d
rhost=
Tue Apr 14 20:30:34 2015 org.apache.qpid.broker:queueDelete user=
qName=qpid.bridge_queue_qpid.tcp:dev3:10089!TAP!TAP!securities.#_4f37e743-b73c-4cc1-80bb-7348a254d36d
rhost=
Tue Apr 14 20:30:36 2015 org.apache.qpid.broker:queueDelete user=
qName=qpid.bridge_queue_qpid.tcp:dev3:10089!TAP!TAP!securities.#_4f37e743-b73c-4cc1-80bb-7348a254d36d
rhost=
Tue Apr 14 20:30:38 2015 org.apache.qpid.broker:queueDelete user=
qName=qpid.bridge_queue_qpid.tcp:dev3:10089!TAP!TAP!securities.#_4f37e743-b73c-4cc1-80bb-7348a254d36d
rhost=
Tue Apr 14 20:30:40 2015 org.apache.qpid.broker:queueDelete user=
qName=qpid.bridge_queue_qpid.tcp:dev3:10089!TAP!TAP!securities.#_4f37e743-b73c-4cc1-80bb-7348a254d36d
rhost=
[5500 more deletes, all 2 seconds apart]

I have a possible explanation for the 2 second cycle. I had a test program
running that was continually opening a connection, publishing a few
thousand messages, and closing the connection. That could be the 2 second
cycle. Or maybe not.

Also, I noticed that the queue is the same for both exchanges. Is this
normal?

-- 
Scott Nicol
snicol@tessellationsoftware.com <sc...@u1.com>
office: (216) 223-8334
mobile: (440) 317-0204