You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@qpid.apache.org by Ted Ross <tr...@redhat.com> on 2023/01/03 13:47:56 UTC

Re: More than configured consumers count

Hi Etka,

Can you tell us how you have configured the qdrouters to act as consumers?
Are you using auto-links or are you using link-routed addresses?

-Ted


On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi
<Ek...@theodpcorp.com.invalid> wrote:

> Hello,
>
> Is this the right mailing DL for QPID related queries.
>
> Just wondering as I have not heard back but given it is a holiday season,
> so I will sit back and wait patiently for someone to reply.
>
> Thanks In Advance.
> Ekta Awasthi
>
> ________________________________
> From: Ekta Awasthi <Ek...@theodpcorp.com>
> Sent: Thursday, December 22, 2022 1:00 AM
> To: users@qpid.apache.org <us...@qpid.apache.org>
> Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh
> Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> Subject: More than configured consumers count
>
> Hi There,
>
> We have few issues with the qpid-dispatch router where we are seeing
> multiple consumers are getting created randomly by our qpid dispatch
> routers. We are unable to trace nor able to replicate this issue in our
> lower env's. Below is the architecture diagram.
>
> I apologize if this isn't the right place to ask this question. In case
> there is another place to communicate, do let me know as this is my first
> time reaching out regarding qpid query😀 Please provide any suggestions or
> feedback to help us understand this issue better. Thanks
>
> Flow:
> Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch
> Routers acting up as consumers in front of brokers (1.16 version) ---> 4
> Broker pairs of activemq(2.18 version) masters and slaves (independent
> pairs)
>
> Problem statement # 1
> We are seeing more than the configured number of consumers count in our
> activemq hawtio console causing messages to sit in delivering count which
> are Un browsable since those messages are currently being delivered to its
> consumers. Having only two qpid dispatch routers(acting as consumers)
> infront of our activemq brokers, the count should always remain 2 but at
> times it is going 3 sometimes 4. To resolve this issue, we are having to
> bounce the qpid’s to release the stuck/bad consumer so that messages can be
> processed/consumed.
>
> Problem statement # 2
> At times we see messages going to delivering count even when there are
> only two configured consumers(2qpids) showing in activemq hawtio console.
> We don't know the cause to why the messages get stuck in delivering count.
> To resolve this, we tried restarting the consumer service but that did not
> help. Next up we tried restarting the brokers, that did not help and
> noticed that all the stuck delivering messages, the broker slowly replayed
> back and Eventually the stuck messages came back to delivering count post
> broker restart. To resolve this issue, we are having to bounce the qpid’s
> to release the messages and that fixes the issue.
>
> Version of qpid-dispatch router
> qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
> qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm
>
>
> Ekta Awasthi
>
> CONFIDENTIALITY NOTICE: The information contained in this email and
> attached document(s) may contain confidential information that is intended
> only for the addressee(s). If you are not the intended recipient, you are
> hereby advised that any disclosure, copying, distribution or the taking of
> any action in reliance upon the information is prohibited. If you have
> received this email in error, please immediately notify the sender and
> delete it from your system.
>

Re: More than configured consumers count

Posted by Ekta Awasthi <Ek...@theodpcorp.com.INVALID>.

Hello Ted,

Thanks for your response.

Is there a slack channel/IM that we can use to chat?

Ekta Awasthi,
Engineer, EAI Operations & Support | Office Depot, Inc.
6600 North Military Trail | Boca Raton, FL 33496-2434
Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com<ma...@officedepot.com>

-- Tips for EAI Support Engagement --
-EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>
-EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support
-As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<ma...@OFFICEDEPOT.COM> or by initiating a MIRT

________________________________
From: Ted Ross <tr...@redhat.com>
Sent: Tuesday, January 3, 2023 5:14 PM
To: Ekta Awasthi <Ek...@theodpcorp.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Thanks for the info.

There should be no more than (the number of configured "in" auto-links) consumers on the set of brokers.  Each "in" (inbound from the router's perspective) link listed in "qdstat -l" represents one consumer.  If your brokers are reporting more consumers than there are "in" links, it's possible that there are other (non-qpid-router) consumers attached to your brokers.

-Ted


On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Regarding Are the od-broker-1-[ms] connections configured via connectors or listeners? ---> They are configured via connectors only below is an example.

#od-broker-1-m connector
connector{
        name: od-broker-1-m
        host: activemq-test-1.odprivatecloud.com<http://activemq-test-1.odprivatecloud.com>
        port: 61618
        role: route-container
        linkCapacity: 40
        sslProfile: od-router-2-test-ssl-profile
        verifyHostName: no  #Since we have cert/hostname mismatches
        #saslMechanisms: PLAIN
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 10:43 AM
To: Ekta Awasthi <Ek...@theodpcorp.com>>
Cc: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Another question...

Are the od-broker-1-[ms] connections configured via connectors or listeners?

-Ted


On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com>> wrote:
Happy New Year Ekta,

Are your micro-services message producers or consumers?

When you see the more-than-expected consumers in Hawtio, get the link status from the routers using "qdstat -l".  There _should_ be one link for each auto-link as long as the targeted broker is reachable.  This should provide some clue as to what is happening.

-Ted


On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Thank you for your response.

We are using AutoLinks for our addresses. Below is an example of a autolink queue in qdrouterd.conf for one activemq broker pair. Thanks

address {
  prefix: test-queue
  waypoint: yes
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: out
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: out
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 8:47 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Hi Etka,

Can you tell us how you have configured the qdrouters to act as consumers?  Are you using auto-links or are you using link-routed addresses?

-Ted


On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi <Ek...@theodpcorp.com.invalid> wrote:
Hello,

Is this the right mailing DL for QPID related queries.

Just wondering as I have not heard back but given it is a holiday season, so I will sit back and wait patiently for someone to reply.

Thanks In Advance.
Ekta Awasthi

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>>
Sent: Thursday, December 22, 2022 1:00 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <ea...@officedepot.com>>
Subject: More than configured consumers count

Hi There,

We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram.

I apologize if this isn't the right place to ask this question. In case there is another place to communicate, do let me know as this is my first time reaching out regarding qpid query😀 Please provide any suggestions or feedback to help us understand this issue better. Thanks

Flow:
Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs)

Problem statement # 1
We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed.

Problem statement # 2
At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue.

Version of qpid-dispatch router
qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm


Ekta Awasthi

CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.

Re: More than configured consumers count

Posted by Ted Ross <tr...@redhat.com>.

Thanks for the info.

There should be no more than (the number of configured "in" auto-links)
consumers on the set of brokers.  Each "in" (inbound from the router's
perspective) link listed in "qdstat -l" represents one consumer.  If your
brokers are reporting more consumers than there are "in" links, it's
possible that there are other (non-qpid-router) consumers attached to your
brokers.

-Ted


On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <Ek...@theodpcorp.com>
wrote:

> Hello Ted,
>
> Regarding Are the od-broker-1-[ms] connections configured via connectors
> or listeners? ---> They are configured via connectors only below is an
> example.
>
> #od-broker-1-m connector
> connector{
>         name: od-broker-1-m
>         host: activemq-test-1.odprivatecloud.com
>         port: 61618
>         role: route-container
>         linkCapacity: 40
>         sslProfile: od-router-2-test-ssl-profile
>         verifyHostName: no  #Since we have cert/hostname mismatches
>         #saslMechanisms: PLAIN
> }
>
> *Ekta Awasthi*,
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Tuesday, January 3, 2023 10:43 AM
> *To:* Ekta Awasthi <Ek...@theodpcorp.com>
> *Cc:* users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade
> (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <
> Nilesh.Khokale@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Another question...
>
> Are the od-broker-1-[ms] connections configured via connectors or
> listeners?
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com> wrote:
>
> Happy New Year Ekta,
>
> Are your micro-services message producers or consumers?
>
> When you see the more-than-expected consumers in Hawtio, get the link
> status from the routers using "qdstat -l".  There _should_ be one link for
> each auto-link as long as the targeted broker is reachable.  This should
> provide some clue as to what is happening.
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>
> wrote:
>
> Hello Ted,
>
> Thank you for your response.
>
> We are using AutoLinks for our addresses. Below is an example of a
> autolink queue in qdrouterd.conf for one activemq broker pair. Thanks
>
> address {
>   prefix: test-queue
>   waypoint: yes
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: out
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: out
> }
>
> *Ekta Awasthi*,
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Tuesday, January 3, 2023 8:47 AM
> *To:* users@qpid.apache.org <us...@qpid.apache.org>
> *Cc:* Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh
> Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Hi Etka,
>
> Can you tell us how you have configured the qdrouters to act as
> consumers?  Are you using auto-links or are you using link-routed addresses?
>
> -Ted
>
>
> On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi
> <Ek...@theodpcorp.com.invalid> wrote:
>
> Hello,
>
> Is this the right mailing DL for QPID related queries.
>
> Just wondering as I have not heard back but given it is a holiday season,
> so I will sit back and wait patiently for someone to reply.
>
> Thanks In Advance.
> Ekta Awasthi
>
> ________________________________
> From: Ekta Awasthi <Ek...@theodpcorp.com>
> Sent: Thursday, December 22, 2022 1:00 AM
> To: users@qpid.apache.org <us...@qpid.apache.org>
> Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh
> Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> Subject: More than configured consumers count
>
> Hi There,
>
> We have few issues with the qpid-dispatch router where we are seeing
> multiple consumers are getting created randomly by our qpid dispatch
> routers. We are unable to trace nor able to replicate this issue in our
> lower env's. Below is the architecture diagram.
>
> I apologize if this isn't the right place to ask this question. In case
> there is another place to communicate, do let me know as this is my first
> time reaching out regarding qpid query😀 Please provide any suggestions or
> feedback to help us understand this issue better. Thanks
>
> Flow:
> Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch
> Routers acting up as consumers in front of brokers (1.16 version) ---> 4
> Broker pairs of activemq(2.18 version) masters and slaves (independent
> pairs)
>
> Problem statement # 1
> We are seeing more than the configured number of consumers count in our
> activemq hawtio console causing messages to sit in delivering count which
> are Un browsable since those messages are currently being delivered to its
> consumers. Having only two qpid dispatch routers(acting as consumers)
> infront of our activemq brokers, the count should always remain 2 but at
> times it is going 3 sometimes 4. To resolve this issue, we are having to
> bounce the qpid’s to release the stuck/bad consumer so that messages can be
> processed/consumed.
>
> Problem statement # 2
> At times we see messages going to delivering count even when there are
> only two configured consumers(2qpids) showing in activemq hawtio console.
> We don't know the cause to why the messages get stuck in delivering count.
> To resolve this, we tried restarting the consumer service but that did not
> help. Next up we tried restarting the brokers, that did not help and
> noticed that all the stuck delivering messages, the broker slowly replayed
> back and Eventually the stuck messages came back to delivering count post
> broker restart. To resolve this issue, we are having to bounce the qpid’s
> to release the messages and that fixes the issue.
>
> Version of qpid-dispatch router
> qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
> qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm
>
>
> Ekta Awasthi
>
> CONFIDENTIALITY NOTICE: The information contained in this email and
> attached document(s) may contain confidential information that is intended
> only for the addressee(s). If you are not the intended recipient, you are
> hereby advised that any disclosure, copying, distribution or the taking of
> any action in reliance upon the information is prohibited. If you have
> received this email in error, please immediately notify the sender and
> delete it from your system.
>
>

Re: More than configured consumers count

Posted by Ted Ross <tr...@redhat.com>.

On Fri, Jan 13, 2023 at 2:21 PM Ekta Awasthi
<Ek...@theodpcorp.com.invalid> wrote:

> Hello Ted,
>
> I did go through both the qpid logs and saw below.
>
> I could see when the second qpid initiated the extra consumer link on
> 1/9/2023 at 4.24 PM EST there were 2775 events stating, "Auto Link
> deactivated" and then shortly after I could see about 5550 events stating
> "Auto Link Activated" That is exactly double the links which second qpid
> activated and therefore we could see two consumer link coming from qpid-2
> and 1 consumer link from qpid-1 for each queue.
>
> Why it activated/created double the links for all the queues is out of my
> mind and that too in the middle of the day when there was no maintenance,
> or anything was happening.
>

Deactivation happens when:
A) The configuration is removed (not the case here)
B) The connection to the broker is dropped

Check that log to see if the deactivations are a result of a dropped
connection to the broker.

Similarly, the activations occur as a result of a new connection being
established with the broker.

Ask yourself: How many auto-links do you expect to have?  How many
connections do you expect to see auto-links on?  Do you see multiple
activations of the same auto-link on the same connection?


>
> Why all of the sudden it deactivated all the links it was attached to? do
> we know what triggered that?
>
> Ekta Awasthi,
> Engineer, EAI Operations & Support | Office Depot, Inc.
> 6600 North Military Trail | Boca Raton, FL 33496-2434
> Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com
> <ma...@officedepot.com>
>
> -- Tips for EAI Support Engagement --
> -EAI Pre-Prod Support: Create requests on the following JIRA board EAI
> Operations Support<
> https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS
> >
> -EAI Production Support: Create requests via IT Service Desk<
> https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service
> portal, instructions click here<
> https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>:
> EAI Support queue --> ODP - Enterprise Apps Integration Support
> -As a reminder, the Service Availability Managers should be engaged for
> any service impacting issues, with a ***Page*** to
> NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<mailto:
> NAITAVAILABILITYMGRS@OFFICEDEPOT.COM> or by initiating a MIRT
>
> ________________________________
> From: Ted Ross <tr...@redhat.com>
> Sent: Thursday, January 12, 2023 4:32 PM
> To: Ekta Awasthi <Ek...@theodpcorp.com>
> Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade
> (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <
> Nilesh.Khokale@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> Subject: Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
>
> Ekta,
>
> Either there's a not-yet discovered (or fixed since 1.16) software defect
> causing this problem that you are seeing or there's some kind of
> misconfiguration in your setup.  If the software requires patching to fix a
> bug, you will need to go through an upgrade to deploy it.
>
> I would suggest looking at the logs from the router(s) that are involved in
> the issues you are seeing to find some clues as to what is going on.
>
> For example, look for the string "Auto Link Activated" in the logs.  This
> is logged every time the router sets up a producer or consumer on a broker
> connection.  You should see exactly one of these for each auto-link you
> have configured whenever a new broker connection is established.  This
> should shed some light on whether there's a configuration issue or a code
> issue.
>
> -Ted
>
>
> On Wed, Jan 11, 2023 at 1:35 PM Ekta Awasthi <Ek...@theodpcorp.com>
> wrote:
>
> > Hello Ted,
> >
> > I agree with you, we can certainly try upgrading but since this issue is
> > happening *directly* in *PROD* and so *frequently*, we need a quick
> > solution to solve these issues. Any help/suggestions will be greatly
> > appreciated.
> >
> > *Ekta Awasthi*,
> >
> > Engineer, EAI Operations & Support | Office Depot, Inc.
> > 6600 North Military Trail | Boca Raton, FL 33496-2434
> > Office: 561-438-3552 | Mobile: 206-966-5577 |
> ekta.awasthi@officedepot.com
> >
> >
> >
> > *-- Tips for EAI Support Engagement --*
> >
> > -EAI Pre-Prod Support: Create requests on the following JIRA board EAI
> > Operations Support
> > <
> https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS
> >
> >
> > -EAI Production Support: Create requests via IT Service Desk
> > <https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service
> > portal, instructions click here
> > <
> https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf
> >:
> > EAI Support queue --> ODP - Enterprise Apps Integration Support
> >
> > -As a reminder, the Service Availability Managers should be engaged for
> > any service impacting issues, with a ***Page*** to
> > NAITAVAILABILITYMGRS@OFFICEDEPOT.COM or by initiating a MIRT
> >
> > ------------------------------
> > *From:* Ted Ross <tr...@redhat.com>
> > *Sent:* Friday, January 6, 2023 3:37 PM
> > *To:* Ekta Awasthi <Ek...@theodpcorp.com>
> > *Cc:* users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade
> > (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale
> (Contractor) <
> > Nilesh.Khokale@officedepot.com>; EAIOpsSupport <
> > eaiopssupport@officedepot.com>
> > *Subject:* Re: More than configured consumers count
> >
> > [CAUTION: EXTERNAL SENDER]
> >
> > Ekta,
> >
> > One thing you should consider is to bring your Qpid Dispatch Router code
> > up to the latest version.  You are running 1.16.0 which is pretty old.
> > Many bugs have been fixed since 1.16.0.
> >
> > -Ted
> >
> >
> > On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <Ekta.Awasthi@theodpcorp.com
> >
> > wrote:
> >
> > Hello Ted,
> >
> > Regarding Are the od-broker-1-[ms] connections configured via connectors
> > or listeners? ---> They are configured via connectors only below is an
> > example.
> >
> > #od-broker-1-m connector
> > connector{
> >         name: od-broker-1-m
> >         host: activemq-test-1.odprivatecloud.com
> >         port: 61618
> >         role: route-container
> >         linkCapacity: 40
> >         sslProfile: od-router-2-test-ssl-profile
> >         verifyHostName: no  #Since we have cert/hostname mismatches
> >         #saslMechanisms: PLAIN
> > }
> >
> > *Ekta Awasthi*,
> >
> > ------------------------------
> > *From:* Ted Ross <tr...@redhat.com>
> > *Sent:* Tuesday, January 3, 2023 10:43 AM
> > *To:* Ekta Awasthi <Ek...@theodpcorp.com>
> > *Cc:* users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade
> > (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale
> (Contractor) <
> > Nilesh.Khokale@officedepot.com>; EAIOpsSupport <
> > eaiopssupport@officedepot.com>
> > *Subject:* Re: More than configured consumers count
> >
> > [CAUTION: EXTERNAL SENDER]
> >
> > Another question...
> >
> > Are the od-broker-1-[ms] connections configured via connectors or
> > listeners?
> >
> > -Ted
> >
> >
> > On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com> wrote:
> >
> > Happy New Year Ekta,
> >
> > Are your micro-services message producers or consumers?
> >
> > When you see the more-than-expected consumers in Hawtio, get the link
> > status from the routers using "qdstat -l".  There _should_ be one link
> for
> > each auto-link as long as the targeted broker is reachable.  This should
> > provide some clue as to what is happening.
> >
> > -Ted
> >
> >
> > On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <
> Ekta.Awasthi@theodpcorp.com>
> > wrote:
> >
> > Hello Ted,
> >
> > Thank you for your response.
> >
> > We are using AutoLinks for our addresses. Below is an example of a
> > autolink queue in qdrouterd.conf for one activemq broker pair. Thanks
> >
> > address {
> >   prefix: test-queue
> >   waypoint: yes
> > }
> >
> > autoLink {
> >   connection: od-broker-1-m
> >   addr: test-queue
> >   dir: in
> > }
> >
> > autoLink {
> >   connection: od-broker-1-m
> >   addr: test-queue
> >   dir: out
> > }
> >
> > autoLink {
> >   connection: od-broker-1-s
> >   addr: test-queue
> >   dir: in
> > }
> >
> > autoLink {
> >   connection: od-broker-1-s
> >   addr: test-queue
> >   dir: out
> > }
> >
> > *Ekta Awasthi*,
> >
> > ------------------------------
> > *From:* Ted Ross <tr...@redhat.com>
> > *Sent:* Tuesday, January 3, 2023 8:47 AM
> > *To:* users@qpid.apache.org <us...@qpid.apache.org>
> > *Cc:* Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh
> > Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <
> > eaiopssupport@officedepot.com>
> > *Subject:* Re: More than configured consumers count
> >
> > [CAUTION: EXTERNAL SENDER]
> >
> > Hi Etka,
> >
> > Can you tell us how you have configured the qdrouters to act as
> > consumers?  Are you using auto-links or are you using link-routed
> addresses?
> >
> > -Ted
> >
> >
> > On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi
> > <Ek...@theodpcorp.com.invalid> wrote:
> >
> > Hello,
> >
> > Is this the right mailing DL for QPID related queries.
> >
> > Just wondering as I have not heard back but given it is a holiday season,
> > so I will sit back and wait patiently for someone to reply.
> >
> > Thanks In Advance.
> > Ekta Awasthi
> >
> > ________________________________
> > From: Ekta Awasthi <Ek...@theodpcorp.com>
> > Sent: Thursday, December 22, 2022 1:00 AM
> > To: users@qpid.apache.org <us...@qpid.apache.org>
> > Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh
> > Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <
> > eaiopssupport@officedepot.com>
> > Subject: More than configured consumers count
> >
> > Hi There,
> >
> > We have few issues with the qpid-dispatch router where we are seeing
> > multiple consumers are getting created randomly by our qpid dispatch
> > routers. We are unable to trace nor able to replicate this issue in our
> > lower env's. Below is the architecture diagram.
> >
> > I apologize if this isn't the right place to ask this question. In case
> > there is another place to communicate, do let me know as this is my first
> > time reaching out regarding qpid query😀 Please provide any suggestions
> or
> > feedback to help us understand this issue better. Thanks
> >
> > Flow:
> > Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch
> > Routers acting up as consumers in front of brokers (1.16 version) ---> 4
> > Broker pairs of activemq(2.18 version) masters and slaves (independent
> > pairs)
> >
> > Problem statement # 1
> > We are seeing more than the configured number of consumers count in our
> > activemq hawtio console causing messages to sit in delivering count which
> > are Un browsable since those messages are currently being delivered to
> its
> > consumers. Having only two qpid dispatch routers(acting as consumers)
> > infront of our activemq brokers, the count should always remain 2 but at
> > times it is going 3 sometimes 4. To resolve this issue, we are having to
> > bounce the qpid’s to release the stuck/bad consumer so that messages can
> be
> > processed/consumed.
> >
> > Problem statement # 2
> > At times we see messages going to delivering count even when there are
> > only two configured consumers(2qpids) showing in activemq hawtio console.
> > We don't know the cause to why the messages get stuck in delivering
> count.
> > To resolve this, we tried restarting the consumer service but that did
> not
> > help. Next up we tried restarting the brokers, that did not help and
> > noticed that all the stuck delivering messages, the broker slowly
> replayed
> > back and Eventually the stuck messages came back to delivering count post
> > broker restart. To resolve this issue, we are having to bounce the qpid’s
> > to release the messages and that fixes the issue.
> >
> > Version of qpid-dispatch router
> > qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
> > qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm
> >
> >
> > Ekta Awasthi
> >
> > CONFIDENTIALITY NOTICE: The information contained in this email and
> > attached document(s) may contain confidential information that is
> intended
> > only for the addressee(s). If you are not the intended recipient, you are
> > hereby advised that any disclosure, copying, distribution or the taking
> of
> > any action in reliance upon the information is prohibited. If you have
> > received this email in error, please immediately notify the sender and
> > delete it from your system.
> >
> >
>
> CONFIDENTIALITY NOTICE: The information contained in this email and
> attached document(s) may contain confidential information that is intended
> only for the addressee(s). If you are not the intended recipient, you are
> hereby advised that any disclosure, copying, distribution or the taking of
> any action in reliance upon the information is prohibited. If you have
> received this email in error, please immediately notify the sender and
> delete it from your system.
>

Re: More than configured consumers count

Posted by Ekta Awasthi <Ek...@theodpcorp.com.INVALID>.

Hello Ted,

I did go through both the qpid logs and saw below.

I could see when the second qpid initiated the extra consumer link on 1/9/2023 at 4.24 PM EST there were 2775 events stating, "Auto Link deactivated" and then shortly after I could see about 5550 events stating "Auto Link Activated" That is exactly double the links which second qpid activated and therefore we could see two consumer link coming from qpid-2 and 1 consumer link from qpid-1 for each queue.

Why it activated/created double the links for all the queues is out of my mind and that too in the middle of the day when there was no maintenance, or anything was happening.

Why all of the sudden it deactivated all the links it was attached to? do we know what triggered that?

Ekta Awasthi,
Engineer, EAI Operations & Support | Office Depot, Inc.
6600 North Military Trail | Boca Raton, FL 33496-2434
Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com<ma...@officedepot.com>

-- Tips for EAI Support Engagement --
-EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>
-EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support
-As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<ma...@OFFICEDEPOT.COM> or by initiating a MIRT

________________________________
From: Ted Ross <tr...@redhat.com>
Sent: Thursday, January 12, 2023 4:32 PM
To: Ekta Awasthi <Ek...@theodpcorp.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]


Ekta,

Either there's a not-yet discovered (or fixed since 1.16) software defect
causing this problem that you are seeing or there's some kind of
misconfiguration in your setup.  If the software requires patching to fix a
bug, you will need to go through an upgrade to deploy it.

I would suggest looking at the logs from the router(s) that are involved in
the issues you are seeing to find some clues as to what is going on.

For example, look for the string "Auto Link Activated" in the logs.  This
is logged every time the router sets up a producer or consumer on a broker
connection.  You should see exactly one of these for each auto-link you
have configured whenever a new broker connection is established.  This
should shed some light on whether there's a configuration issue or a code
issue.

-Ted


On Wed, Jan 11, 2023 at 1:35 PM Ekta Awasthi <Ek...@theodpcorp.com>
wrote:

> Hello Ted,
>
> I agree with you, we can certainly try upgrading but since this issue is
> happening *directly* in *PROD* and so *frequently*, we need a quick
> solution to solve these issues. Any help/suggestions will be greatly
> appreciated.
>
> *Ekta Awasthi*,
>
> Engineer, EAI Operations & Support | Office Depot, Inc.
> 6600 North Military Trail | Boca Raton, FL 33496-2434
> Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com
>
>
>
> *-- Tips for EAI Support Engagement --*
>
> -EAI Pre-Prod Support: Create requests on the following JIRA board EAI
> Operations Support
> <https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>
>
> -EAI Production Support: Create requests via IT Service Desk
> <https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service
> portal, instructions click here
> <https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>:
> EAI Support queue --> ODP - Enterprise Apps Integration Support
>
> -As a reminder, the Service Availability Managers should be engaged for
> any service impacting issues, with a ***Page*** to
> NAITAVAILABILITYMGRS@OFFICEDEPOT.COM or by initiating a MIRT
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Friday, January 6, 2023 3:37 PM
> *To:* Ekta Awasthi <Ek...@theodpcorp.com>
> *Cc:* users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade
> (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <
> Nilesh.Khokale@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Ekta,
>
> One thing you should consider is to bring your Qpid Dispatch Router code
> up to the latest version.  You are running 1.16.0 which is pretty old.
> Many bugs have been fixed since 1.16.0.
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <Ek...@theodpcorp.com>
> wrote:
>
> Hello Ted,
>
> Regarding Are the od-broker-1-[ms] connections configured via connectors
> or listeners? ---> They are configured via connectors only below is an
> example.
>
> #od-broker-1-m connector
> connector{
>         name: od-broker-1-m
>         host: activemq-test-1.odprivatecloud.com
>         port: 61618
>         role: route-container
>         linkCapacity: 40
>         sslProfile: od-router-2-test-ssl-profile
>         verifyHostName: no  #Since we have cert/hostname mismatches
>         #saslMechanisms: PLAIN
> }
>
> *Ekta Awasthi*,
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Tuesday, January 3, 2023 10:43 AM
> *To:* Ekta Awasthi <Ek...@theodpcorp.com>
> *Cc:* users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade
> (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <
> Nilesh.Khokale@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Another question...
>
> Are the od-broker-1-[ms] connections configured via connectors or
> listeners?
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com> wrote:
>
> Happy New Year Ekta,
>
> Are your micro-services message producers or consumers?
>
> When you see the more-than-expected consumers in Hawtio, get the link
> status from the routers using "qdstat -l".  There _should_ be one link for
> each auto-link as long as the targeted broker is reachable.  This should
> provide some clue as to what is happening.
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>
> wrote:
>
> Hello Ted,
>
> Thank you for your response.
>
> We are using AutoLinks for our addresses. Below is an example of a
> autolink queue in qdrouterd.conf for one activemq broker pair. Thanks
>
> address {
>   prefix: test-queue
>   waypoint: yes
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: out
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: out
> }
>
> *Ekta Awasthi*,
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Tuesday, January 3, 2023 8:47 AM
> *To:* users@qpid.apache.org <us...@qpid.apache.org>
> *Cc:* Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh
> Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Hi Etka,
>
> Can you tell us how you have configured the qdrouters to act as
> consumers?  Are you using auto-links or are you using link-routed addresses?
>
> -Ted
>
>
> On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi
> <Ek...@theodpcorp.com.invalid> wrote:
>
> Hello,
>
> Is this the right mailing DL for QPID related queries.
>
> Just wondering as I have not heard back but given it is a holiday season,
> so I will sit back and wait patiently for someone to reply.
>
> Thanks In Advance.
> Ekta Awasthi
>
> ________________________________
> From: Ekta Awasthi <Ek...@theodpcorp.com>
> Sent: Thursday, December 22, 2022 1:00 AM
> To: users@qpid.apache.org <us...@qpid.apache.org>
> Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh
> Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> Subject: More than configured consumers count
>
> Hi There,
>
> We have few issues with the qpid-dispatch router where we are seeing
> multiple consumers are getting created randomly by our qpid dispatch
> routers. We are unable to trace nor able to replicate this issue in our
> lower env's. Below is the architecture diagram.
>
> I apologize if this isn't the right place to ask this question. In case
> there is another place to communicate, do let me know as this is my first
> time reaching out regarding qpid query😀 Please provide any suggestions or
> feedback to help us understand this issue better. Thanks
>
> Flow:
> Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch
> Routers acting up as consumers in front of brokers (1.16 version) ---> 4
> Broker pairs of activemq(2.18 version) masters and slaves (independent
> pairs)
>
> Problem statement # 1
> We are seeing more than the configured number of consumers count in our
> activemq hawtio console causing messages to sit in delivering count which
> are Un browsable since those messages are currently being delivered to its
> consumers. Having only two qpid dispatch routers(acting as consumers)
> infront of our activemq brokers, the count should always remain 2 but at
> times it is going 3 sometimes 4. To resolve this issue, we are having to
> bounce the qpid’s to release the stuck/bad consumer so that messages can be
> processed/consumed.
>
> Problem statement # 2
> At times we see messages going to delivering count even when there are
> only two configured consumers(2qpids) showing in activemq hawtio console.
> We don't know the cause to why the messages get stuck in delivering count.
> To resolve this, we tried restarting the consumer service but that did not
> help. Next up we tried restarting the brokers, that did not help and
> noticed that all the stuck delivering messages, the broker slowly replayed
> back and Eventually the stuck messages came back to delivering count post
> broker restart. To resolve this issue, we are having to bounce the qpid’s
> to release the messages and that fixes the issue.
>
> Version of qpid-dispatch router
> qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
> qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm
>
>
> Ekta Awasthi
>
> CONFIDENTIALITY NOTICE: The information contained in this email and
> attached document(s) may contain confidential information that is intended
> only for the addressee(s). If you are not the intended recipient, you are
> hereby advised that any disclosure, copying, distribution or the taking of
> any action in reliance upon the information is prohibited. If you have
> received this email in error, please immediately notify the sender and
> delete it from your system.
>
>

CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.

Re: More than configured consumers count

Posted by Ted Ross <tr...@redhat.com>.

Ekta,

Either there's a not-yet discovered (or fixed since 1.16) software defect
causing this problem that you are seeing or there's some kind of
misconfiguration in your setup.  If the software requires patching to fix a
bug, you will need to go through an upgrade to deploy it.

I would suggest looking at the logs from the router(s) that are involved in
the issues you are seeing to find some clues as to what is going on.

For example, look for the string "Auto Link Activated" in the logs.  This
is logged every time the router sets up a producer or consumer on a broker
connection.  You should see exactly one of these for each auto-link you
have configured whenever a new broker connection is established.  This
should shed some light on whether there's a configuration issue or a code
issue.

-Ted


On Wed, Jan 11, 2023 at 1:35 PM Ekta Awasthi <Ek...@theodpcorp.com>
wrote:

> Hello Ted,
>
> I agree with you, we can certainly try upgrading but since this issue is
> happening *directly* in *PROD* and so *frequently*, we need a quick
> solution to solve these issues. Any help/suggestions will be greatly
> appreciated.
>
> *Ekta Awasthi*,
>
> Engineer, EAI Operations & Support | Office Depot, Inc.
> 6600 North Military Trail | Boca Raton, FL 33496-2434
> Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com
>
>
>
> *-- Tips for EAI Support Engagement --*
>
> -EAI Pre-Prod Support: Create requests on the following JIRA board EAI
> Operations Support
> <https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>
>
> -EAI Production Support: Create requests via IT Service Desk
> <https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service
> portal, instructions click here
> <https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>:
> EAI Support queue --> ODP - Enterprise Apps Integration Support
>
> -As a reminder, the Service Availability Managers should be engaged for
> any service impacting issues, with a ***Page*** to
> NAITAVAILABILITYMGRS@OFFICEDEPOT.COM or by initiating a MIRT
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Friday, January 6, 2023 3:37 PM
> *To:* Ekta Awasthi <Ek...@theodpcorp.com>
> *Cc:* users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade
> (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <
> Nilesh.Khokale@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Ekta,
>
> One thing you should consider is to bring your Qpid Dispatch Router code
> up to the latest version.  You are running 1.16.0 which is pretty old.
> Many bugs have been fixed since 1.16.0.
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <Ek...@theodpcorp.com>
> wrote:
>
> Hello Ted,
>
> Regarding Are the od-broker-1-[ms] connections configured via connectors
> or listeners? ---> They are configured via connectors only below is an
> example.
>
> #od-broker-1-m connector
> connector{
>         name: od-broker-1-m
>         host: activemq-test-1.odprivatecloud.com
>         port: 61618
>         role: route-container
>         linkCapacity: 40
>         sslProfile: od-router-2-test-ssl-profile
>         verifyHostName: no  #Since we have cert/hostname mismatches
>         #saslMechanisms: PLAIN
> }
>
> *Ekta Awasthi*,
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Tuesday, January 3, 2023 10:43 AM
> *To:* Ekta Awasthi <Ek...@theodpcorp.com>
> *Cc:* users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade
> (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <
> Nilesh.Khokale@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Another question...
>
> Are the od-broker-1-[ms] connections configured via connectors or
> listeners?
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com> wrote:
>
> Happy New Year Ekta,
>
> Are your micro-services message producers or consumers?
>
> When you see the more-than-expected consumers in Hawtio, get the link
> status from the routers using "qdstat -l".  There _should_ be one link for
> each auto-link as long as the targeted broker is reachable.  This should
> provide some clue as to what is happening.
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>
> wrote:
>
> Hello Ted,
>
> Thank you for your response.
>
> We are using AutoLinks for our addresses. Below is an example of a
> autolink queue in qdrouterd.conf for one activemq broker pair. Thanks
>
> address {
>   prefix: test-queue
>   waypoint: yes
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: out
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: out
> }
>
> *Ekta Awasthi*,
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Tuesday, January 3, 2023 8:47 AM
> *To:* users@qpid.apache.org <us...@qpid.apache.org>
> *Cc:* Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh
> Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Hi Etka,
>
> Can you tell us how you have configured the qdrouters to act as
> consumers?  Are you using auto-links or are you using link-routed addresses?
>
> -Ted
>
>
> On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi
> <Ek...@theodpcorp.com.invalid> wrote:
>
> Hello,
>
> Is this the right mailing DL for QPID related queries.
>
> Just wondering as I have not heard back but given it is a holiday season,
> so I will sit back and wait patiently for someone to reply.
>
> Thanks In Advance.
> Ekta Awasthi
>
> ________________________________
> From: Ekta Awasthi <Ek...@theodpcorp.com>
> Sent: Thursday, December 22, 2022 1:00 AM
> To: users@qpid.apache.org <us...@qpid.apache.org>
> Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh
> Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> Subject: More than configured consumers count
>
> Hi There,
>
> We have few issues with the qpid-dispatch router where we are seeing
> multiple consumers are getting created randomly by our qpid dispatch
> routers. We are unable to trace nor able to replicate this issue in our
> lower env's. Below is the architecture diagram.
>
> I apologize if this isn't the right place to ask this question. In case
> there is another place to communicate, do let me know as this is my first
> time reaching out regarding qpid query😀 Please provide any suggestions or
> feedback to help us understand this issue better. Thanks
>
> Flow:
> Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch
> Routers acting up as consumers in front of brokers (1.16 version) ---> 4
> Broker pairs of activemq(2.18 version) masters and slaves (independent
> pairs)
>
> Problem statement # 1
> We are seeing more than the configured number of consumers count in our
> activemq hawtio console causing messages to sit in delivering count which
> are Un browsable since those messages are currently being delivered to its
> consumers. Having only two qpid dispatch routers(acting as consumers)
> infront of our activemq brokers, the count should always remain 2 but at
> times it is going 3 sometimes 4. To resolve this issue, we are having to
> bounce the qpid’s to release the stuck/bad consumer so that messages can be
> processed/consumed.
>
> Problem statement # 2
> At times we see messages going to delivering count even when there are
> only two configured consumers(2qpids) showing in activemq hawtio console.
> We don't know the cause to why the messages get stuck in delivering count.
> To resolve this, we tried restarting the consumer service but that did not
> help. Next up we tried restarting the brokers, that did not help and
> noticed that all the stuck delivering messages, the broker slowly replayed
> back and Eventually the stuck messages came back to delivering count post
> broker restart. To resolve this issue, we are having to bounce the qpid’s
> to release the messages and that fixes the issue.
>
> Version of qpid-dispatch router
> qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
> qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm
>
>
> Ekta Awasthi
>
> CONFIDENTIALITY NOTICE: The information contained in this email and
> attached document(s) may contain confidential information that is intended
> only for the addressee(s). If you are not the intended recipient, you are
> hereby advised that any disclosure, copying, distribution or the taking of
> any action in reliance upon the information is prohibited. If you have
> received this email in error, please immediately notify the sender and
> delete it from your system.
>
>

Re: More than configured consumers count

Posted by Ekta Awasthi <Ek...@theodpcorp.com.INVALID>.

Hello Ted and All,

Any update / suggestions?

Ekta Awasthi,
Engineer, EAI Operations & Support | Office Depot, Inc.
6600 North Military Trail | Boca Raton, FL 33496-2434
Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com<ma...@officedepot.com>

-- Tips for EAI Support Engagement --
-EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>
-EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support
-As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<ma...@OFFICEDEPOT.COM> or by initiating a MIRT

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>
Sent: Wednesday, January 11, 2023 1:39 PM
To: Ted Ross <tr...@redhat.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

I have also created a qpid ticket on QPID JIRA Board.

https://issues.apache.org/jira/browse/DISPATCH-2353


Ekta Awasthi,

Engineer, EAI Operations & Support | Office Depot, Inc.
6600 North Military Trail | Boca Raton, FL 33496-2434
Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com<ma...@officedepot.com>



-- Tips for EAI Support Engagement --

-EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>

-EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support

-As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<ma...@OFFICEDEPOT.COM> or by initiating a MIRT

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>
Sent: Wednesday, January 11, 2023 1:35 PM
To: Ted Ross <tr...@redhat.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

Hello Ted,

I agree with you, we can certainly try upgrading but since this issue is happening directly in PROD and so frequently, we need a quick solution to solve these issues. Any help/suggestions will be greatly appreciated.


Ekta Awasthi,

Engineer, EAI Operations & Support | Office Depot, Inc.
6600 North Military Trail | Boca Raton, FL 33496-2434
Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com<ma...@officedepot.com>



-- Tips for EAI Support Engagement --

-EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>

-EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support

-As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<ma...@OFFICEDEPOT.COM> or by initiating a MIRT

________________________________
From: Ted Ross <tr...@redhat.com>
Sent: Friday, January 6, 2023 3:37 PM
To: Ekta Awasthi <Ek...@theodpcorp.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Ekta,

One thing you should consider is to bring your Qpid Dispatch Router code up to the latest version.  You are running 1.16.0 which is pretty old.  Many bugs have been fixed since 1.16.0.

-Ted


On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Regarding Are the od-broker-1-[ms] connections configured via connectors or listeners? ---> They are configured via connectors only below is an example.

#od-broker-1-m connector
connector{
        name: od-broker-1-m
        host: activemq-test-1.odprivatecloud.com<http://activemq-test-1.odprivatecloud.com>
        port: 61618
        role: route-container
        linkCapacity: 40
        sslProfile: od-router-2-test-ssl-profile
        verifyHostName: no  #Since we have cert/hostname mismatches
        #saslMechanisms: PLAIN
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 10:43 AM
To: Ekta Awasthi <Ek...@theodpcorp.com>>
Cc: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Another question...

Are the od-broker-1-[ms] connections configured via connectors or listeners?

-Ted


On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com>> wrote:
Happy New Year Ekta,

Are your micro-services message producers or consumers?

When you see the more-than-expected consumers in Hawtio, get the link status from the routers using "qdstat -l".  There _should_ be one link for each auto-link as long as the targeted broker is reachable.  This should provide some clue as to what is happening.

-Ted


On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Thank you for your response.

We are using AutoLinks for our addresses. Below is an example of a autolink queue in qdrouterd.conf for one activemq broker pair. Thanks

address {
  prefix: test-queue
  waypoint: yes
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: out
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: out
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 8:47 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Hi Etka,

Can you tell us how you have configured the qdrouters to act as consumers?  Are you using auto-links or are you using link-routed addresses?

-Ted


On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi <Ek...@theodpcorp.com.invalid> wrote:
Hello,

Is this the right mailing DL for QPID related queries.

Just wondering as I have not heard back but given it is a holiday season, so I will sit back and wait patiently for someone to reply.

Thanks In Advance.
Ekta Awasthi

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>>
Sent: Thursday, December 22, 2022 1:00 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <ea...@officedepot.com>>
Subject: More than configured consumers count

Hi There,

We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram.

I apologize if this isn't the right place to ask this question. In case there is another place to communicate, do let me know as this is my first time reaching out regarding qpid query😀 Please provide any suggestions or feedback to help us understand this issue better. Thanks

Flow:
Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs)

Problem statement # 1
We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed.

Problem statement # 2
At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue.

Version of qpid-dispatch router
qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm


Ekta Awasthi

CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.

Re: More than configured consumers count

Posted by Ekta Awasthi <Ek...@theodpcorp.com.INVALID>.

I have also created a qpid ticket on QPID JIRA Board.

https://issues.apache.org/jira/browse/DISPATCH-2353

Ekta Awasthi,
Engineer, EAI Operations & Support | Office Depot, Inc.
6600 North Military Trail | Boca Raton, FL 33496-2434
Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com<ma...@officedepot.com>

-- Tips for EAI Support Engagement --
-EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>
-EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support
-As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<ma...@OFFICEDEPOT.COM> or by initiating a MIRT

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>
Sent: Wednesday, January 11, 2023 1:35 PM
To: Ted Ross <tr...@redhat.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

Hello Ted,

I agree with you, we can certainly try upgrading but since this issue is happening directly in PROD and so frequently, we need a quick solution to solve these issues. Any help/suggestions will be greatly appreciated.


Ekta Awasthi,

Engineer, EAI Operations & Support | Office Depot, Inc.
6600 North Military Trail | Boca Raton, FL 33496-2434
Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com<ma...@officedepot.com>



-- Tips for EAI Support Engagement --

-EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>

-EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support

-As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<ma...@OFFICEDEPOT.COM> or by initiating a MIRT

________________________________
From: Ted Ross <tr...@redhat.com>
Sent: Friday, January 6, 2023 3:37 PM
To: Ekta Awasthi <Ek...@theodpcorp.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Ekta,

One thing you should consider is to bring your Qpid Dispatch Router code up to the latest version.  You are running 1.16.0 which is pretty old.  Many bugs have been fixed since 1.16.0.

-Ted


On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Regarding Are the od-broker-1-[ms] connections configured via connectors or listeners? ---> They are configured via connectors only below is an example.

#od-broker-1-m connector
connector{
        name: od-broker-1-m
        host: activemq-test-1.odprivatecloud.com<http://activemq-test-1.odprivatecloud.com>
        port: 61618
        role: route-container
        linkCapacity: 40
        sslProfile: od-router-2-test-ssl-profile
        verifyHostName: no  #Since we have cert/hostname mismatches
        #saslMechanisms: PLAIN
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 10:43 AM
To: Ekta Awasthi <Ek...@theodpcorp.com>>
Cc: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Another question...

Are the od-broker-1-[ms] connections configured via connectors or listeners?

-Ted


On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com>> wrote:
Happy New Year Ekta,

Are your micro-services message producers or consumers?

When you see the more-than-expected consumers in Hawtio, get the link status from the routers using "qdstat -l".  There _should_ be one link for each auto-link as long as the targeted broker is reachable.  This should provide some clue as to what is happening.

-Ted


On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Thank you for your response.

We are using AutoLinks for our addresses. Below is an example of a autolink queue in qdrouterd.conf for one activemq broker pair. Thanks

address {
  prefix: test-queue
  waypoint: yes
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: out
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: out
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 8:47 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Hi Etka,

Can you tell us how you have configured the qdrouters to act as consumers?  Are you using auto-links or are you using link-routed addresses?

-Ted


On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi <Ek...@theodpcorp.com.invalid> wrote:
Hello,

Is this the right mailing DL for QPID related queries.

Just wondering as I have not heard back but given it is a holiday season, so I will sit back and wait patiently for someone to reply.

Thanks In Advance.
Ekta Awasthi

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>>
Sent: Thursday, December 22, 2022 1:00 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <ea...@officedepot.com>>
Subject: More than configured consumers count

Hi There,

We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram.

I apologize if this isn't the right place to ask this question. In case there is another place to communicate, do let me know as this is my first time reaching out regarding qpid query😀 Please provide any suggestions or feedback to help us understand this issue better. Thanks

Flow:
Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs)

Problem statement # 1
We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed.

Problem statement # 2
At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue.

Version of qpid-dispatch router
qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm


Ekta Awasthi

CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.

Re: More than configured consumers count

Posted by Ekta Awasthi <Ek...@theodpcorp.com.INVALID>.

Hello Ted,

I agree with you, we can certainly try upgrading but since this issue is happening directly in PROD and so frequently, we need a quick solution to solve these issues. Any help/suggestions will be greatly appreciated.

Ekta Awasthi,
Engineer, EAI Operations & Support | Office Depot, Inc.
6600 North Military Trail | Boca Raton, FL 33496-2434
Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com<ma...@officedepot.com>

-- Tips for EAI Support Engagement --
-EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>
-EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support
-As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<ma...@OFFICEDEPOT.COM> or by initiating a MIRT

________________________________
From: Ted Ross <tr...@redhat.com>
Sent: Friday, January 6, 2023 3:37 PM
To: Ekta Awasthi <Ek...@theodpcorp.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Ekta,

One thing you should consider is to bring your Qpid Dispatch Router code up to the latest version.  You are running 1.16.0 which is pretty old.  Many bugs have been fixed since 1.16.0.

-Ted


On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Regarding Are the od-broker-1-[ms] connections configured via connectors or listeners? ---> They are configured via connectors only below is an example.

#od-broker-1-m connector
connector{
        name: od-broker-1-m
        host: activemq-test-1.odprivatecloud.com<http://activemq-test-1.odprivatecloud.com>
        port: 61618
        role: route-container
        linkCapacity: 40
        sslProfile: od-router-2-test-ssl-profile
        verifyHostName: no  #Since we have cert/hostname mismatches
        #saslMechanisms: PLAIN
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 10:43 AM
To: Ekta Awasthi <Ek...@theodpcorp.com>>
Cc: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Another question...

Are the od-broker-1-[ms] connections configured via connectors or listeners?

-Ted


On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com>> wrote:
Happy New Year Ekta,

Are your micro-services message producers or consumers?

When you see the more-than-expected consumers in Hawtio, get the link status from the routers using "qdstat -l".  There _should_ be one link for each auto-link as long as the targeted broker is reachable.  This should provide some clue as to what is happening.

-Ted


On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Thank you for your response.

We are using AutoLinks for our addresses. Below is an example of a autolink queue in qdrouterd.conf for one activemq broker pair. Thanks

address {
  prefix: test-queue
  waypoint: yes
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: out
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: out
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 8:47 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Hi Etka,

Can you tell us how you have configured the qdrouters to act as consumers?  Are you using auto-links or are you using link-routed addresses?

-Ted


On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi <Ek...@theodpcorp.com.invalid> wrote:
Hello,

Is this the right mailing DL for QPID related queries.

Just wondering as I have not heard back but given it is a holiday season, so I will sit back and wait patiently for someone to reply.

Thanks In Advance.
Ekta Awasthi

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>>
Sent: Thursday, December 22, 2022 1:00 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <ea...@officedepot.com>>
Subject: More than configured consumers count

Hi There,

We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram.

I apologize if this isn't the right place to ask this question. In case there is another place to communicate, do let me know as this is my first time reaching out regarding qpid query😀 Please provide any suggestions or feedback to help us understand this issue better. Thanks

Flow:
Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs)

Problem statement # 1
We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed.

Problem statement # 2
At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue.

Version of qpid-dispatch router
qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm


Ekta Awasthi

CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.

Re: More than configured consumers count

Posted by Ted Ross <tr...@redhat.com>.

Ekta,

One thing you should consider is to bring your Qpid Dispatch Router code up
to the latest version.  You are running 1.16.0 which is pretty old.  Many
bugs have been fixed since 1.16.0.

-Ted


On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <Ek...@theodpcorp.com>
wrote:

> Hello Ted,
>
> Regarding Are the od-broker-1-[ms] connections configured via connectors
> or listeners? ---> They are configured via connectors only below is an
> example.
>
> #od-broker-1-m connector
> connector{
>         name: od-broker-1-m
>         host: activemq-test-1.odprivatecloud.com
>         port: 61618
>         role: route-container
>         linkCapacity: 40
>         sslProfile: od-router-2-test-ssl-profile
>         verifyHostName: no  #Since we have cert/hostname mismatches
>         #saslMechanisms: PLAIN
> }
>
> *Ekta Awasthi*,
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Tuesday, January 3, 2023 10:43 AM
> *To:* Ekta Awasthi <Ek...@theodpcorp.com>
> *Cc:* users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade
> (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <
> Nilesh.Khokale@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Another question...
>
> Are the od-broker-1-[ms] connections configured via connectors or
> listeners?
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com> wrote:
>
> Happy New Year Ekta,
>
> Are your micro-services message producers or consumers?
>
> When you see the more-than-expected consumers in Hawtio, get the link
> status from the routers using "qdstat -l".  There _should_ be one link for
> each auto-link as long as the targeted broker is reachable.  This should
> provide some clue as to what is happening.
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>
> wrote:
>
> Hello Ted,
>
> Thank you for your response.
>
> We are using AutoLinks for our addresses. Below is an example of a
> autolink queue in qdrouterd.conf for one activemq broker pair. Thanks
>
> address {
>   prefix: test-queue
>   waypoint: yes
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: out
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: out
> }
>
> *Ekta Awasthi*,
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Tuesday, January 3, 2023 8:47 AM
> *To:* users@qpid.apache.org <us...@qpid.apache.org>
> *Cc:* Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh
> Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Hi Etka,
>
> Can you tell us how you have configured the qdrouters to act as
> consumers?  Are you using auto-links or are you using link-routed addresses?
>
> -Ted
>
>
> On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi
> <Ek...@theodpcorp.com.invalid> wrote:
>
> Hello,
>
> Is this the right mailing DL for QPID related queries.
>
> Just wondering as I have not heard back but given it is a holiday season,
> so I will sit back and wait patiently for someone to reply.
>
> Thanks In Advance.
> Ekta Awasthi
>
> ________________________________
> From: Ekta Awasthi <Ek...@theodpcorp.com>
> Sent: Thursday, December 22, 2022 1:00 AM
> To: users@qpid.apache.org <us...@qpid.apache.org>
> Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh
> Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> Subject: More than configured consumers count
>
> Hi There,
>
> We have few issues with the qpid-dispatch router where we are seeing
> multiple consumers are getting created randomly by our qpid dispatch
> routers. We are unable to trace nor able to replicate this issue in our
> lower env's. Below is the architecture diagram.
>
> I apologize if this isn't the right place to ask this question. In case
> there is another place to communicate, do let me know as this is my first
> time reaching out regarding qpid query😀 Please provide any suggestions or
> feedback to help us understand this issue better. Thanks
>
> Flow:
> Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch
> Routers acting up as consumers in front of brokers (1.16 version) ---> 4
> Broker pairs of activemq(2.18 version) masters and slaves (independent
> pairs)
>
> Problem statement # 1
> We are seeing more than the configured number of consumers count in our
> activemq hawtio console causing messages to sit in delivering count which
> are Un browsable since those messages are currently being delivered to its
> consumers. Having only two qpid dispatch routers(acting as consumers)
> infront of our activemq brokers, the count should always remain 2 but at
> times it is going 3 sometimes 4. To resolve this issue, we are having to
> bounce the qpid’s to release the stuck/bad consumer so that messages can be
> processed/consumed.
>
> Problem statement # 2
> At times we see messages going to delivering count even when there are
> only two configured consumers(2qpids) showing in activemq hawtio console.
> We don't know the cause to why the messages get stuck in delivering count.
> To resolve this, we tried restarting the consumer service but that did not
> help. Next up we tried restarting the brokers, that did not help and
> noticed that all the stuck delivering messages, the broker slowly replayed
> back and Eventually the stuck messages came back to delivering count post
> broker restart. To resolve this issue, we are having to bounce the qpid’s
> to release the messages and that fixes the issue.
>
> Version of qpid-dispatch router
> qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
> qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm
>
>
> Ekta Awasthi
>
> CONFIDENTIALITY NOTICE: The information contained in this email and
> attached document(s) may contain confidential information that is intended
> only for the addressee(s). If you are not the intended recipient, you are
> hereby advised that any disclosure, copying, distribution or the taking of
> any action in reliance upon the information is prohibited. If you have
> received this email in error, please immediately notify the sender and
> delete it from your system.
>
>

Re: More than configured consumers count

Posted by Ekta Awasthi <Ek...@theodpcorp.com.INVALID>.

Hello Ted,

Regarding Are the od-broker-1-[ms] connections configured via connectors or listeners? ---> They are configured via connectors only below is an example.

#od-broker-1-m connector
connector{
        name: od-broker-1-m
        host: activemq-test-1.odprivatecloud.com
        port: 61618
        role: route-container
        linkCapacity: 40
        sslProfile: od-router-2-test-ssl-profile
        verifyHostName: no  #Since we have cert/hostname mismatches
        #saslMechanisms: PLAIN
}

Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>
Sent: Tuesday, January 3, 2023 10:43 AM
To: Ekta Awasthi <Ek...@theodpcorp.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Another question...

Are the od-broker-1-[ms] connections configured via connectors or listeners?

-Ted


On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com>> wrote:
Happy New Year Ekta,

Are your micro-services message producers or consumers?

When you see the more-than-expected consumers in Hawtio, get the link status from the routers using "qdstat -l".  There _should_ be one link for each auto-link as long as the targeted broker is reachable.  This should provide some clue as to what is happening.

-Ted


On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Thank you for your response.

We are using AutoLinks for our addresses. Below is an example of a autolink queue in qdrouterd.conf for one activemq broker pair. Thanks

address {
  prefix: test-queue
  waypoint: yes
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: out
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: out
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 8:47 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Hi Etka,

Can you tell us how you have configured the qdrouters to act as consumers?  Are you using auto-links or are you using link-routed addresses?

-Ted


On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi <Ek...@theodpcorp.com.invalid> wrote:
Hello,

Is this the right mailing DL for QPID related queries.

Just wondering as I have not heard back but given it is a holiday season, so I will sit back and wait patiently for someone to reply.

Thanks In Advance.
Ekta Awasthi

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>>
Sent: Thursday, December 22, 2022 1:00 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <ea...@officedepot.com>>
Subject: More than configured consumers count

Hi There,

We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram.

I apologize if this isn't the right place to ask this question. In case there is another place to communicate, do let me know as this is my first time reaching out regarding qpid query😀 Please provide any suggestions or feedback to help us understand this issue better. Thanks

Flow:
Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs)

Problem statement # 1
We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed.

Problem statement # 2
At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue.

Version of qpid-dispatch router
qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm


Ekta Awasthi

CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.

Re: More than configured consumers count

Posted by Ekta Awasthi <Ek...@theodpcorp.com.INVALID>.

Happy New Year to you also.

Are your micro-services message producers or consumers? ---> We have microservices acting as producers as well as consumers, it all depends.

When you see the more-than-expected consumers in Hawtio, get the link status from the routers using "qdstat -l".  There _should_ be one link for each auto-link as long as the targeted broker is reachable.  This should provide some clue as to what is happening.  ----> I think this number looks about ok.

Ekta Awasthi,
Engineer, EAI Operations & Support | Office Depot, Inc.
6600 North Military Trail | Boca Raton, FL 33496-2434
Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awasthi@officedepot.com<ma...@officedepot.com>

-- Tips for EAI Support Engagement --
-EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS>
-EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support
-As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to NAITAVAILABILITYMGRS@OFFICEDEPOT.COM<ma...@OFFICEDEPOT.COM> or by initiating a MIRT

________________________________
From: Ted Ross <tr...@redhat.com>
Sent: Tuesday, January 3, 2023 10:43 AM
To: Ekta Awasthi <Ek...@theodpcorp.com>
Cc: users@qpid.apache.org <us...@qpid.apache.org>; Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Another question...

Are the od-broker-1-[ms] connections configured via connectors or listeners?

-Ted


On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com>> wrote:
Happy New Year Ekta,

Are your micro-services message producers or consumers?

When you see the more-than-expected consumers in Hawtio, get the link status from the routers using "qdstat -l".  There _should_ be one link for each auto-link as long as the targeted broker is reachable.  This should provide some clue as to what is happening.

-Ted


On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>> wrote:
Hello Ted,

Thank you for your response.

We are using AutoLinks for our addresses. Below is an example of a autolink queue in qdrouterd.conf for one activemq broker pair. Thanks

address {
  prefix: test-queue
  waypoint: yes
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: out
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: out
}


Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>>
Sent: Tuesday, January 3, 2023 8:47 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@officedepot.com>>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>>; EAIOpsSupport <ea...@officedepot.com>>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Hi Etka,

Can you tell us how you have configured the qdrouters to act as consumers?  Are you using auto-links or are you using link-routed addresses?

-Ted


On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi <Ek...@theodpcorp.com.invalid> wrote:
Hello,

Is this the right mailing DL for QPID related queries.

Just wondering as I have not heard back but given it is a holiday season, so I will sit back and wait patiently for someone to reply.

Thanks In Advance.
Ekta Awasthi

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>>
Sent: Thursday, December 22, 2022 1:00 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <ea...@officedepot.com>>
Subject: More than configured consumers count

Hi There,

We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram.

I apologize if this isn't the right place to ask this question. In case there is another place to communicate, do let me know as this is my first time reaching out regarding qpid query😀 Please provide any suggestions or feedback to help us understand this issue better. Thanks

Flow:
Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs)

Problem statement # 1
We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed.

Problem statement # 2
At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue.

Version of qpid-dispatch router
qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm


Ekta Awasthi

CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.

Re: More than configured consumers count

Posted by Ted Ross <tr...@redhat.com>.

Another question...

Are the od-broker-1-[ms] connections configured via connectors or listeners?

-Ted


On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com> wrote:

> Happy New Year Ekta,
>
> Are your micro-services message producers or consumers?
>
> When you see the more-than-expected consumers in Hawtio, get the link
> status from the routers using "qdstat -l".  There _should_ be one link for
> each auto-link as long as the targeted broker is reachable.  This should
> provide some clue as to what is happening.
>
> -Ted
>
>
> On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>
> wrote:
>
>> Hello Ted,
>>
>> Thank you for your response.
>>
>> We are using AutoLinks for our addresses. Below is an example of a
>> autolink queue in qdrouterd.conf for one activemq broker pair. Thanks
>>
>> address {
>>   prefix: test-queue
>>   waypoint: yes
>> }
>>
>> autoLink {
>>   connection: od-broker-1-m
>>   addr: test-queue
>>   dir: in
>> }
>>
>> autoLink {
>>   connection: od-broker-1-m
>>   addr: test-queue
>>   dir: out
>> }
>>
>> autoLink {
>>   connection: od-broker-1-s
>>   addr: test-queue
>>   dir: in
>> }
>>
>> autoLink {
>>   connection: od-broker-1-s
>>   addr: test-queue
>>   dir: out
>> }
>>
>> *Ekta Awasthi*,
>>
>> ------------------------------
>> *From:* Ted Ross <tr...@redhat.com>
>> *Sent:* Tuesday, January 3, 2023 8:47 AM
>> *To:* users@qpid.apache.org <us...@qpid.apache.org>
>> *Cc:* Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh
>> Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <
>> eaiopssupport@officedepot.com>
>> *Subject:* Re: More than configured consumers count
>>
>> [CAUTION: EXTERNAL SENDER]
>>
>> Hi Etka,
>>
>> Can you tell us how you have configured the qdrouters to act as
>> consumers?  Are you using auto-links or are you using link-routed addresses?
>>
>> -Ted
>>
>>
>> On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi
>> <Ek...@theodpcorp.com.invalid> wrote:
>>
>> Hello,
>>
>> Is this the right mailing DL for QPID related queries.
>>
>> Just wondering as I have not heard back but given it is a holiday season,
>> so I will sit back and wait patiently for someone to reply.
>>
>> Thanks In Advance.
>> Ekta Awasthi
>>
>> ________________________________
>> From: Ekta Awasthi <Ek...@theodpcorp.com>
>> Sent: Thursday, December 22, 2022 1:00 AM
>> To: users@qpid.apache.org <us...@qpid.apache.org>
>> Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh
>> Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <
>> eaiopssupport@officedepot.com>
>> Subject: More than configured consumers count
>>
>> Hi There,
>>
>> We have few issues with the qpid-dispatch router where we are seeing
>> multiple consumers are getting created randomly by our qpid dispatch
>> routers. We are unable to trace nor able to replicate this issue in our
>> lower env's. Below is the architecture diagram.
>>
>> I apologize if this isn't the right place to ask this question. In case
>> there is another place to communicate, do let me know as this is my first
>> time reaching out regarding qpid query😀 Please provide any suggestions or
>> feedback to help us understand this issue better. Thanks
>>
>> Flow:
>> Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch
>> Routers acting up as consumers in front of brokers (1.16 version) ---> 4
>> Broker pairs of activemq(2.18 version) masters and slaves (independent
>> pairs)
>>
>> Problem statement # 1
>> We are seeing more than the configured number of consumers count in our
>> activemq hawtio console causing messages to sit in delivering count which
>> are Un browsable since those messages are currently being delivered to its
>> consumers. Having only two qpid dispatch routers(acting as consumers)
>> infront of our activemq brokers, the count should always remain 2 but at
>> times it is going 3 sometimes 4. To resolve this issue, we are having to
>> bounce the qpid’s to release the stuck/bad consumer so that messages can be
>> processed/consumed.
>>
>> Problem statement # 2
>> At times we see messages going to delivering count even when there are
>> only two configured consumers(2qpids) showing in activemq hawtio console.
>> We don't know the cause to why the messages get stuck in delivering count.
>> To resolve this, we tried restarting the consumer service but that did not
>> help. Next up we tried restarting the brokers, that did not help and
>> noticed that all the stuck delivering messages, the broker slowly replayed
>> back and Eventually the stuck messages came back to delivering count post
>> broker restart. To resolve this issue, we are having to bounce the qpid’s
>> to release the messages and that fixes the issue.
>>
>> Version of qpid-dispatch router
>> qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
>> qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm
>>
>>
>> Ekta Awasthi
>>
>> CONFIDENTIALITY NOTICE: The information contained in this email and
>> attached document(s) may contain confidential information that is intended
>> only for the addressee(s). If you are not the intended recipient, you are
>> hereby advised that any disclosure, copying, distribution or the taking of
>> any action in reliance upon the information is prohibited. If you have
>> received this email in error, please immediately notify the sender and
>> delete it from your system.
>>
>>

Re: More than configured consumers count

Posted by Ted Ross <tr...@redhat.com>.

Happy New Year Ekta,

Are your micro-services message producers or consumers?

When you see the more-than-expected consumers in Hawtio, get the link
status from the routers using "qdstat -l".  There _should_ be one link for
each auto-link as long as the targeted broker is reachable.  This should
provide some clue as to what is happening.

-Ted


On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <Ek...@theodpcorp.com>
wrote:

> Hello Ted,
>
> Thank you for your response.
>
> We are using AutoLinks for our addresses. Below is an example of a
> autolink queue in qdrouterd.conf for one activemq broker pair. Thanks
>
> address {
>   prefix: test-queue
>   waypoint: yes
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-m
>   addr: test-queue
>   dir: out
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: in
> }
>
> autoLink {
>   connection: od-broker-1-s
>   addr: test-queue
>   dir: out
> }
>
> *Ekta Awasthi*,
>
> ------------------------------
> *From:* Ted Ross <tr...@redhat.com>
> *Sent:* Tuesday, January 3, 2023 8:47 AM
> *To:* users@qpid.apache.org <us...@qpid.apache.org>
> *Cc:* Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh
> Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> *Subject:* Re: More than configured consumers count
>
> [CAUTION: EXTERNAL SENDER]
>
> Hi Etka,
>
> Can you tell us how you have configured the qdrouters to act as
> consumers?  Are you using auto-links or are you using link-routed addresses?
>
> -Ted
>
>
> On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi
> <Ek...@theodpcorp.com.invalid> wrote:
>
> Hello,
>
> Is this the right mailing DL for QPID related queries.
>
> Just wondering as I have not heard back but given it is a holiday season,
> so I will sit back and wait patiently for someone to reply.
>
> Thanks In Advance.
> Ekta Awasthi
>
> ________________________________
> From: Ekta Awasthi <Ek...@theodpcorp.com>
> Sent: Thursday, December 22, 2022 1:00 AM
> To: users@qpid.apache.org <us...@qpid.apache.org>
> Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh
> Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <
> eaiopssupport@officedepot.com>
> Subject: More than configured consumers count
>
> Hi There,
>
> We have few issues with the qpid-dispatch router where we are seeing
> multiple consumers are getting created randomly by our qpid dispatch
> routers. We are unable to trace nor able to replicate this issue in our
> lower env's. Below is the architecture diagram.
>
> I apologize if this isn't the right place to ask this question. In case
> there is another place to communicate, do let me know as this is my first
> time reaching out regarding qpid query😀 Please provide any suggestions or
> feedback to help us understand this issue better. Thanks
>
> Flow:
> Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch
> Routers acting up as consumers in front of brokers (1.16 version) ---> 4
> Broker pairs of activemq(2.18 version) masters and slaves (independent
> pairs)
>
> Problem statement # 1
> We are seeing more than the configured number of consumers count in our
> activemq hawtio console causing messages to sit in delivering count which
> are Un browsable since those messages are currently being delivered to its
> consumers. Having only two qpid dispatch routers(acting as consumers)
> infront of our activemq brokers, the count should always remain 2 but at
> times it is going 3 sometimes 4. To resolve this issue, we are having to
> bounce the qpid’s to release the stuck/bad consumer so that messages can be
> processed/consumed.
>
> Problem statement # 2
> At times we see messages going to delivering count even when there are
> only two configured consumers(2qpids) showing in activemq hawtio console.
> We don't know the cause to why the messages get stuck in delivering count.
> To resolve this, we tried restarting the consumer service but that did not
> help. Next up we tried restarting the brokers, that did not help and
> noticed that all the stuck delivering messages, the broker slowly replayed
> back and Eventually the stuck messages came back to delivering count post
> broker restart. To resolve this issue, we are having to bounce the qpid’s
> to release the messages and that fixes the issue.
>
> Version of qpid-dispatch router
> qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
> qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm
>
>
> Ekta Awasthi
>
> CONFIDENTIALITY NOTICE: The information contained in this email and
> attached document(s) may contain confidential information that is intended
> only for the addressee(s). If you are not the intended recipient, you are
> hereby advised that any disclosure, copying, distribution or the taking of
> any action in reliance upon the information is prohibited. If you have
> received this email in error, please immediately notify the sender and
> delete it from your system.
>
>

Re: More than configured consumers count

Posted by Ekta Awasthi <Ek...@theodpcorp.com.INVALID>.

Hello Ted,

Thank you for your response.

We are using AutoLinks for our addresses. Below is an example of a autolink queue in qdrouterd.conf for one activemq broker pair. Thanks

address {
  prefix: test-queue
  waypoint: yes
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-m
  addr: test-queue
  dir: out
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: in
}

autoLink {
  connection: od-broker-1-s
  addr: test-queue
  dir: out
}

Ekta Awasthi,

________________________________
From: Ted Ross <tr...@redhat.com>
Sent: Tuesday, January 3, 2023 8:47 AM
To: users@qpid.apache.org <us...@qpid.apache.org>
Cc: Ajit Tathawade (CompuCom) <Aj...@officedepot.com>; Nilesh Khokale (Contractor) <Ni...@officedepot.com>; EAIOpsSupport <ea...@officedepot.com>
Subject: Re: More than configured consumers count

[CAUTION: EXTERNAL SENDER]

Hi Etka,

Can you tell us how you have configured the qdrouters to act as consumers?  Are you using auto-links or are you using link-routed addresses?

-Ted


On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi <Ek...@theodpcorp.com.invalid> wrote:
Hello,

Is this the right mailing DL for QPID related queries.

Just wondering as I have not heard back but given it is a holiday season, so I will sit back and wait patiently for someone to reply.

Thanks In Advance.
Ekta Awasthi

________________________________
From: Ekta Awasthi <Ek...@theodpcorp.com>>
Sent: Thursday, December 22, 2022 1:00 AM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Cc: Ajit Tathawade (CompuCom) <Aj...@OfficeDepot.com>; Nilesh Khokale (Contractor) <Ni...@OfficeDepot.com>; EAIOpsSupport <ea...@officedepot.com>>
Subject: More than configured consumers count

Hi There,

We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram.

I apologize if this isn't the right place to ask this question. In case there is another place to communicate, do let me know as this is my first time reaching out regarding qpid query😀 Please provide any suggestions or feedback to help us understand this issue better. Thanks

Flow:
Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs)

Problem statement # 1
We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed.

Problem statement # 2
At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue.

Version of qpid-dispatch router
qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm


Ekta Awasthi

CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.