You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Marcel Meulemans <m....@tkhinnovations.com> on 2018/04/11 19:14:39 UTC
Qpid dispatch unstable inter-router connections
Hi all,
I am running a three node fully connected mesh of dispatch routers with
10000 attached clients and I am seeing some unstable inter-router
connections (I am sending around 1000 message per second through the
network). The inter-router connections fail every so many seconds with the
message: Connection to router-2:55672 failed: amqp:session:invalid-field
sequencing error, expected delivery-id 7, got 6
I am not sure if this is to be expected / unexpected or that I have a
faulty configuration.
The routers are connected as follows: router-0 -> router-1, router-0 ->
router-2, router-1 -> router-2
Attached are my qdrouter.conf (from one of the routers) and a log snippet
from router-0 at debug level from connection drop to connection
re-established to connection drop again.
Versions: qpid-dispatch@1.0.1-rc1, qpid-proton@0.20.0
Hope someone can help.
Cheers,
Marcel
Re: Qpid dispatch unstable inter-router connections
Posted by Matthieu Simonin <ma...@inria.fr>.
Hi Marcel,
Funny coïncidence ;)
I ran also some load tests lately, I haven't seen this behaviour using
the same 3 fully connected nodes topology.
I was using qpid-dispatch@1.0.1 and qpid-proton@0.21
My workload consisted in sending RPCs[1] through the mesh.
- 10K clients / 20 servers
- clients send at 1msg/s that leads to 10K msg/s on the bus for
"fire-and-forget" RPCs and thus 20K msg/s for "request/reply" style RPCs.
Note that the above
- uses small message size (approx 1KB)
- the load lasted only 5 min but was repeated with different number of clients.
Thus routers were up for 3 hours.
- clients connections were balanced on the 3 routers.
I'm afraid I can't help further but I'll be although interested in the
discussion.
Best,
[1]: https://docs.openstack.org/oslo.messaging/latest/
Matthieu
----- Mail original -----
> De: "Marcel Meulemans" <m....@tkhinnovations.com>
> À: "users" <us...@qpid.apache.org>
> Envoyé: Mercredi 11 Avril 2018 21:14:39
> Objet: Qpid dispatch unstable inter-router connections
> Hi all,
>
> I am running a three node fully connected mesh of dispatch routers with
> 10000 attached clients and I am seeing some unstable inter-router
> connections (I am sending around 1000 message per second through the
> network). The inter-router connections fail every so many seconds with the
> message: Connection to router-2:55672 failed: amqp:session:invalid-field
> sequencing error, expected delivery-id 7, got 6
>
> I am not sure if this is to be expected / unexpected or that I have a
> faulty configuration.
>
> The routers are connected as follows: router-0 -> router-1, router-0 ->
> router-2, router-1 -> router-2
> Attached are my qdrouter.conf (from one of the routers) and a log snippet
> from router-0 at debug level from connection drop to connection
> re-established to connection drop again.
> Versions: qpid-dispatch@1.0.1-rc1, qpid-proton@0.20.0
>
> Hope someone can help.
>
> Cheers,
> Marcel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Qpid dispatch unstable inter-router connections
Posted by Marcel Meulemans <m....@tkhinnovations.com>.
On Wed, Apr 11, 2018 at 11:21 PM, Ganesh Murthy <gm...@redhat.com> wrote:
>
> > message: Connection to router-2:55672 failed: amqp:session:invalid-field
> > sequencing error, expected delivery-id 7, got 6
> >
> Is it complaining about the same delivery id (7 got 6) every time in the
> sequencing error ? I am hoping not.
No, the id's are different every time.
--
Marcel
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Qpid dispatch unstable inter-router connections
Posted by Ganesh Murthy <gm...@redhat.com>.
On Wed, Apr 11, 2018 at 3:14 PM, Marcel Meulemans <
m.meulemans@tkhinnovations.com> wrote:
> Hi all,
>
> I am running a three node fully connected mesh of dispatch routers with
> 10000 attached clients and I am seeing some unstable inter-router
> connections (I am sending around 1000 message per second through the
> network). The inter-router connections fail every so many seconds with the
> message: Connection to router-2:55672 failed: amqp:session:invalid-field
> sequencing error, expected delivery-id 7, got 6
>
Is it complaining about the same delivery id (7 got 6) every time in the
sequencing error ? I am hoping not.
>
> I am not sure if this is to be expected / unexpected or that I have a
> faulty configuration.
>
> The routers are connected as follows: router-0 -> router-1, router-0 ->
> router-2, router-1 -> router-2
> Attached are my qdrouter.conf (from one of the routers) and a log snippet
> from router-0 at debug level from connection drop to connection
> re-established to connection drop again.
> Versions: qpid-dispatch@1.0.1-rc1, qpid-proton@0.20.0
>
> Hope someone can help.
>
> Cheers,
> Marcel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
Re: Qpid dispatch unstable inter-router connections
Posted by Marcel Meulemans <m....@tkhinnovations.com>.
On Wed, Apr 11, 2018 at 11:47 PM, Ted Ross <tr...@redhat.com> wrote:
> This is a pretty low-level framing issue that definitely should not be
> happening. Also, your configuration should be fine. Is there
> anything unusual about the network over which the inter-router
> connections are communicating?
I think not, the routers are all running on separate google compute
engine machines on the same network.
> Are you sending large messages?
Nope, messages are less than 1K
> It might be difficult, but if you can get a packet capture of an
> inter-router connection during such a failure, that would be very
> helpful. It looks like the expected delivery-id was bumped
> prematurely or delivery 6 arrived twice.
No problem, attached to the issue below. Actually the capture shows
something interesting. Just before every inter router connection drop
there is a rejected disposition saying:
Condition: qd:forbidden
Description: Deliveries to a multicast address must be pre-settled
> It will be easier for us to track this issue if you raise a Jira at
> https://issues.apache.org for this problem.
Sure: https://issues.apache.org/jira/browse/DISPATCH-966
--
Marcel
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Qpid dispatch unstable inter-router connections
Posted by Ted Ross <tr...@redhat.com>.
Hi Marcel,
This is a pretty low-level framing issue that definitely should not be
happening. Also, your configuration should be fine. Is there
anything unusual about the network over which the inter-router
connections are communicating? Are you sending large messages?
It might be difficult, but if you can get a packet capture of an
inter-router connection during such a failure, that would be very
helpful. It looks like the expected delivery-id was bumped
prematurely or delivery 6 arrived twice.
It will be easier for us to track this issue if you raise a Jira at
https://issues.apache.org for this problem.
-Ted
On Wed, Apr 11, 2018 at 3:14 PM, Marcel Meulemans
<m....@tkhinnovations.com> wrote:
> Hi all,
>
> I am running a three node fully connected mesh of dispatch routers with
> 10000 attached clients and I am seeing some unstable inter-router
> connections (I am sending around 1000 message per second through the
> network). The inter-router connections fail every so many seconds with the
> message: Connection to router-2:55672 failed: amqp:session:invalid-field
> sequencing error, expected delivery-id 7, got 6
>
> I am not sure if this is to be expected / unexpected or that I have a faulty
> configuration.
>
> The routers are connected as follows: router-0 -> router-1, router-0 ->
> router-2, router-1 -> router-2
> Attached are my qdrouter.conf (from one of the routers) and a log snippet
> from router-0 at debug level from connection drop to connection
> re-established to connection drop again.
> Versions: qpid-dispatch@1.0.1-rc1, qpid-proton@0.20.0
>
> Hope someone can help.
>
> Cheers,
> Marcel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org