You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Kim van der Riet (Jira)" <ji...@apache.org> on 2020/03/10 14:55:00 UTC
[jira] [Assigned] (QPID-8425) [qpid-cpp] Channel leak on federation
links
[ https://issues.apache.org/jira/browse/QPID-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kim van der Riet reassigned QPID-8425:
--------------------------------------
Assignee: Kim van der Riet
> [qpid-cpp] Channel leak on federation links
> -------------------------------------------
>
> Key: QPID-8425
> URL: https://issues.apache.org/jira/browse/QPID-8425
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Reporter: Kim van der Riet
> Assignee: Kim van der Riet
> Priority: Major
> Attachments: BZ1748054.patch, channel_test.sh
>
>
> If a broker is repeatedly killed and restarted when it is federated to another broker, the second broker will run out of channels with a "channel pool is empty" message. The channel being used for the federation link is not being returned to the channel pool, and eventually (after ~32k restarts), the channel pool becomes exhausted.
> A reproducer is contained in the attached file "channel_test.sh". If a small change is made to the Link.cpp constructor which shrinks the channel pool to 5 (see the diff below), then this test will show the error without having to wait hours for ~32k restarts. The test does 10 restarts.
> {noformat}
> diff --git a/src/qpid/broker/Link.cpp b/src/qpid/broker/Link.cpp
> index 14737e730..790c8ac5e 100644
> --- a/src/qpid/broker/Link.cpp
> +++ b/src/qpid/broker/Link.cpp
> @@ -149,7 +149,7 @@ Link::Link(const string& _name,
> currentInterval(1),
> reconnectNext(0), // Index of next address for reconnecting in url.
> nextFreeChannel(1),
> - freeChannels(1, framing::CHANNEL_MAX),
> + freeChannels(1, 6),
> connection(0),
> agent(0),
> listener(l),
> {noformat}
> Running the test with the above temporary patch, the following is observed:
> {noformat}
> Channel allocations from broker 6001:
> 3377:2020-03-06 13:29:36 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 1
> 3538:2020-03-06 13:29:36 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 2
> 3739:2020-03-06 13:29:39 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 3
> 3934:2020-03-06 13:29:43 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 4
> 4022:2020-03-06 13:29:47 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 5
> 4110:2020-03-06 13:29:51 [System] debug Exception constructed: Link qpid.tcp:localhost:5001 channel pool is empty
> 4111:2020-03-06 13:29:51 [System] error Link qpid.tcp:localhost:5001 channel pool is empty
> 4154:2020-03-06 13:29:52 [System] debug Exception constructed: Link qpid.tcp:localhost:5001 channel pool is empty
> 4155:2020-03-06 13:29:52 [System] error Link qpid.tcp:localhost:5001 channel pool is empty
> 4253:2020-03-06 13:29:55 [System] debug Exception constructed: Link qpid.tcp:localhost:5001 channel pool is empty
> 4254:2020-03-06 13:29:55 [System] error Link qpid.tcp:localhost:5001 channel pool is empty
> 4297:2020-03-06 13:29:56 [System] debug Exception constructed: Link qpid.tcp:localhost:5001 channel pool is empty
> 4298:2020-03-06 13:29:56 [System] error Link qpid.tcp:localhost:5001 channel pool is empty
> ...
> (repeated several times more)
> {noformat}
>
> A fix which returns the channels on links that are closing to the channel pool is suggested in attached patch BZ1748054.patch. With this patch applied (together with the temporary pool-shrink patch above), the following is now observed:
> {noformat}
> Channel allocations from broker 6001:
> 2020-03-06 12:20:59 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 1
> 2020-03-06 12:20:59 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 2
> 2020-03-06 12:21:02 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 3
> 2020-03-06 12:21:06 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 4
> 2020-03-06 12:21:10 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 5
> 2020-03-06 12:21:14 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 1
> 2020-03-06 12:21:18 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 3
> 2020-03-06 12:21:22 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 4
> 2020-03-06 12:21:26 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 5
> 2020-03-06 12:21:30 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 1
> 2020-03-06 12:21:34 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 3
> 2020-03-06 12:21:38 [Broker] debug Link qpid.tcp:localhost:5001 allocates channel: 4
> {noformat}
> Channel 2 is used by a bridge which remains open (qpid.bridge_session_amq.fanout). The channels are re-used in a cyclical pattern, which I think is the intention.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org