Message appears to be sent but not received

[Lionel van den Berg <li...@gmail.com>]

Another kind of vague one. We have found from our application logs at site
that some messages that we are sending appear to be sent OK on one end but
are never received by the subscriber.

This seems to coincide with topics where the traffic volume is low, does
anyone have any pointers on where to look first?

Re: Message appears to be sent but not received

[Lionel van den Berg <li...@gmail.com>]

I’m glad it was my code too, and expected it to be. At least now I have
some additional ideas for debugging I’m the future.

On Mon, 9 Apr 2018 at 12:17 pm, Tim Bain <tb...@alumni.duke.edu> wrote:

> I was hoping you'd say that, because if the broker was randomly failing to
> deliver a couple of messages out of a huge number, that wasn't going to be
> a lot of fun to track down. So I'm glad you figured it out, for both our
> sakes.
>
> Tim
>
> On Sun, Apr 8, 2018, 6:58 PM Lionel van den Berg <li...@gmail.com>
> wrote:
>
>> Good news, it’s our bug. We have some logic that is automatically
>> cleaning up inactive consumers, that logic was supposed to be turned off.
>>
>>
>> On Mon, 9 Apr 2018 at 12:39 am, Tim Bain <tb...@gmail.com> wrote:
>>
>>> The number of producers and the message volume should both be irrelevant.
>>> Though of course if there's a bug, all bets are off.
>>>
>>> On Apr 8, 2018 5:53 AM, "Lionel van den Berg" <li...@gmail.com> wrote:
>>>
>>> I’m going to try to get someone to try your suggestions at site tomorrow.
>>>
>>> Regarding your question, yes, the subscriber is a server that is started
>>> before the producers and never restarted. Does it matter that there are
>>> multiple producers to the topic. Also it’s a very low volume topic, there
>>> is lots of data on other topics on the same connection.
>>>
>>>
>>>
>>>
>>> On Sun, 8 Apr 2018 at 1:43 am, Tim Bain <tb...@alumni.duke.edu> wrote:
>>>
>>> > I'd suggest you use a JMX viewer such as JConsole, and navigate through
>>> the
>>> > org.apache.activemq MBeans to find the subscription in question. On the
>>> > Attributes page under the subscription, you'll see counts for the
>>> number
>>> of
>>> > messages enqueued and dequeued.
>>> >
>>> > If those don't match, it means your messages are not getting delivered
>>> to
>>> > the subscriber. (Could they be expiring before they can be consumed?)
>>> >
>>> > If they match, but are lower than the number you sent to the topic
>>> while
>>> > the consumer was subscribed, we'll have to dig deeper.
>>> >
>>> > Also, I understand from your response that these consumers have been
>>> > subscribed and connected the entire time the messages were being sent.
>>> Is
>>> > that what you're saying?
>>> >
>>> > Tim
>>> >
>>> > On Fri, Apr 6, 2018 at 7:11 AM, Lionel van den Berg <lionelv@gmail.com
>>> >
>>> > wrote:
>>> >
>>> > > Non-durable an using the same connection as other topics that are
>>> still
>>> > > working. No filters.
>>> > >
>>> > >
>>> > > On Fri, 6 Apr 2018 at 11:03 pm, Tim Bain <tb...@alumni.duke.edu>
>>> wrote:
>>> > >
>>> > > > Are the topic subscriptions durable or non-durable? If the latter,
>>> were
>>> > > the
>>> > > > clients disconnected at the time the messages in question were
>>> sent?
>>> > > >
>>> > > > Also, do the subscribers in question use any selectors?
>>> > > >
>>> > > > Tim
>>> > > >
>>> > > > On Fri, Apr 6, 2018, 12:26 AM Lionel van den Berg <
>>> lionelv@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Another kind of vague one. We have found from our application
>>> logs
>>> at
>>> > > > site
>>> > > > > that some messages that we are sending appear to be sent OK on
>>> one
>>> > end
>>> > > > but
>>> > > > > are never received by the subscriber.
>>> > > > >
>>> > > > > This seems to coincide with topics where the traffic volume is
>>> low,
>>> > > does
>>> > > > > anyone have any pointers on where to look first?
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>

Re: Message appears to be sent but not received

[Tim Bain <tb...@alumni.duke.edu>]

I was hoping you'd say that, because if the broker was randomly failing to
deliver a couple of messages out of a huge number, that wasn't going to be
a lot of fun to track down. So I'm glad you figured it out, for both our
sakes.

Tim

On Sun, Apr 8, 2018, 6:58 PM Lionel van den Berg <li...@gmail.com> wrote:

> Good news, it’s our bug. We have some logic that is automatically cleaning
> up inactive consumers, that logic was supposed to be turned off.
>
>
> On Mon, 9 Apr 2018 at 12:39 am, Tim Bain <tb...@gmail.com> wrote:
>
>> The number of producers and the message volume should both be irrelevant.
>> Though of course if there's a bug, all bets are off.
>>
>> On Apr 8, 2018 5:53 AM, "Lionel van den Berg" <li...@gmail.com> wrote:
>>
>> I’m going to try to get someone to try your suggestions at site tomorrow.
>>
>> Regarding your question, yes, the subscriber is a server that is started
>> before the producers and never restarted. Does it matter that there are
>> multiple producers to the topic. Also it’s a very low volume topic, there
>> is lots of data on other topics on the same connection.
>>
>>
>>
>>
>> On Sun, 8 Apr 2018 at 1:43 am, Tim Bain <tb...@alumni.duke.edu> wrote:
>>
>> > I'd suggest you use a JMX viewer such as JConsole, and navigate through
>> the
>> > org.apache.activemq MBeans to find the subscription in question. On the
>> > Attributes page under the subscription, you'll see counts for the number
>> of
>> > messages enqueued and dequeued.
>> >
>> > If those don't match, it means your messages are not getting delivered
>> to
>> > the subscriber. (Could they be expiring before they can be consumed?)
>> >
>> > If they match, but are lower than the number you sent to the topic while
>> > the consumer was subscribed, we'll have to dig deeper.
>> >
>> > Also, I understand from your response that these consumers have been
>> > subscribed and connected the entire time the messages were being sent.
>> Is
>> > that what you're saying?
>> >
>> > Tim
>> >
>> > On Fri, Apr 6, 2018 at 7:11 AM, Lionel van den Berg <li...@gmail.com>
>> > wrote:
>> >
>> > > Non-durable an using the same connection as other topics that are
>> still
>> > > working. No filters.
>> > >
>> > >
>> > > On Fri, 6 Apr 2018 at 11:03 pm, Tim Bain <tb...@alumni.duke.edu>
>> wrote:
>> > >
>> > > > Are the topic subscriptions durable or non-durable? If the latter,
>> were
>> > > the
>> > > > clients disconnected at the time the messages in question were sent?
>> > > >
>> > > > Also, do the subscribers in question use any selectors?
>> > > >
>> > > > Tim
>> > > >
>> > > > On Fri, Apr 6, 2018, 12:26 AM Lionel van den Berg <
>> lionelv@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Another kind of vague one. We have found from our application logs
>> at
>> > > > site
>> > > > > that some messages that we are sending appear to be sent OK on one
>> > end
>> > > > but
>> > > > > are never received by the subscriber.
>> > > > >
>> > > > > This seems to coincide with topics where the traffic volume is
>> low,
>> > > does
>> > > > > anyone have any pointers on where to look first?
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: Message appears to be sent but not received

[Lionel van den Berg <li...@gmail.com>]

Good news, it’s our bug. We have some logic that is automatically cleaning
up inactive consumers, that logic was supposed to be turned off.


On Mon, 9 Apr 2018 at 12:39 am, Tim Bain <tb...@gmail.com> wrote:

> The number of producers and the message volume should both be irrelevant.
> Though of course if there's a bug, all bets are off.
>
> On Apr 8, 2018 5:53 AM, "Lionel van den Berg" <li...@gmail.com> wrote:
>
> I’m going to try to get someone to try your suggestions at site tomorrow.
>
> Regarding your question, yes, the subscriber is a server that is started
> before the producers and never restarted. Does it matter that there are
> multiple producers to the topic. Also it’s a very low volume topic, there
> is lots of data on other topics on the same connection.
>
>
>
>
> On Sun, 8 Apr 2018 at 1:43 am, Tim Bain <tb...@alumni.duke.edu> wrote:
>
> > I'd suggest you use a JMX viewer such as JConsole, and navigate through
> the
> > org.apache.activemq MBeans to find the subscription in question. On the
> > Attributes page under the subscription, you'll see counts for the number
> of
> > messages enqueued and dequeued.
> >
> > If those don't match, it means your messages are not getting delivered to
> > the subscriber. (Could they be expiring before they can be consumed?)
> >
> > If they match, but are lower than the number you sent to the topic while
> > the consumer was subscribed, we'll have to dig deeper.
> >
> > Also, I understand from your response that these consumers have been
> > subscribed and connected the entire time the messages were being sent. Is
> > that what you're saying?
> >
> > Tim
> >
> > On Fri, Apr 6, 2018 at 7:11 AM, Lionel van den Berg <li...@gmail.com>
> > wrote:
> >
> > > Non-durable an using the same connection as other topics that are still
> > > working. No filters.
> > >
> > >
> > > On Fri, 6 Apr 2018 at 11:03 pm, Tim Bain <tb...@alumni.duke.edu>
> wrote:
> > >
> > > > Are the topic subscriptions durable or non-durable? If the latter,
> were
> > > the
> > > > clients disconnected at the time the messages in question were sent?
> > > >
> > > > Also, do the subscribers in question use any selectors?
> > > >
> > > > Tim
> > > >
> > > > On Fri, Apr 6, 2018, 12:26 AM Lionel van den Berg <lionelv@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Another kind of vague one. We have found from our application logs
> at
> > > > site
> > > > > that some messages that we are sending appear to be sent OK on one
> > end
> > > > but
> > > > > are never received by the subscriber.
> > > > >
> > > > > This seems to coincide with topics where the traffic volume is low,
> > > does
> > > > > anyone have any pointers on where to look first?
> > > > >
> > > >
> > >
> >
>

Re: Message appears to be sent but not received

[Tim Bain <tb...@gmail.com>]

The number of producers and the message volume should both be irrelevant.
Though of course if there's a bug, all bets are off.

On Apr 8, 2018 5:53 AM, "Lionel van den Berg" <li...@gmail.com> wrote:

I’m going to try to get someone to try your suggestions at site tomorrow.

Regarding your question, yes, the subscriber is a server that is started
before the producers and never restarted. Does it matter that there are
multiple producers to the topic. Also it’s a very low volume topic, there
is lots of data on other topics on the same connection.




On Sun, 8 Apr 2018 at 1:43 am, Tim Bain <tb...@alumni.duke.edu> wrote:

> I'd suggest you use a JMX viewer such as JConsole, and navigate through
the
> org.apache.activemq MBeans to find the subscription in question. On the
> Attributes page under the subscription, you'll see counts for the number
of
> messages enqueued and dequeued.
>
> If those don't match, it means your messages are not getting delivered to
> the subscriber. (Could they be expiring before they can be consumed?)
>
> If they match, but are lower than the number you sent to the topic while
> the consumer was subscribed, we'll have to dig deeper.
>
> Also, I understand from your response that these consumers have been
> subscribed and connected the entire time the messages were being sent. Is
> that what you're saying?
>
> Tim
>
> On Fri, Apr 6, 2018 at 7:11 AM, Lionel van den Berg <li...@gmail.com>
> wrote:
>
> > Non-durable an using the same connection as other topics that are still
> > working. No filters.
> >
> >
> > On Fri, 6 Apr 2018 at 11:03 pm, Tim Bain <tb...@alumni.duke.edu> wrote:
> >
> > > Are the topic subscriptions durable or non-durable? If the latter,
were
> > the
> > > clients disconnected at the time the messages in question were sent?
> > >
> > > Also, do the subscribers in question use any selectors?
> > >
> > > Tim
> > >
> > > On Fri, Apr 6, 2018, 12:26 AM Lionel van den Berg <li...@gmail.com>
> > > wrote:
> > >
> > > > Another kind of vague one. We have found from our application logs
at
> > > site
> > > > that some messages that we are sending appear to be sent OK on one
> end
> > > but
> > > > are never received by the subscriber.
> > > >
> > > > This seems to coincide with topics where the traffic volume is low,
> > does
> > > > anyone have any pointers on where to look first?
> > > >
> > >
> >
>

Re: Message appears to be sent but not received

[Lionel van den Berg <li...@gmail.com>]

I’m going to try to get someone to try your suggestions at site tomorrow.

Regarding your question, yes, the subscriber is a server that is started
before the producers and never restarted. Does it matter that there are
multiple producers to the topic. Also it’s a very low volume topic, there
is lots of data on other topics on the same connection.



On Sun, 8 Apr 2018 at 1:43 am, Tim Bain <tb...@alumni.duke.edu> wrote:

> I'd suggest you use a JMX viewer such as JConsole, and navigate through the
> org.apache.activemq MBeans to find the subscription in question. On the
> Attributes page under the subscription, you'll see counts for the number of
> messages enqueued and dequeued.
>
> If those don't match, it means your messages are not getting delivered to
> the subscriber. (Could they be expiring before they can be consumed?)
>
> If they match, but are lower than the number you sent to the topic while
> the consumer was subscribed, we'll have to dig deeper.
>
> Also, I understand from your response that these consumers have been
> subscribed and connected the entire time the messages were being sent. Is
> that what you're saying?
>
> Tim
>
> On Fri, Apr 6, 2018 at 7:11 AM, Lionel van den Berg <li...@gmail.com>
> wrote:
>
> > Non-durable an using the same connection as other topics that are still
> > working. No filters.
> >
> >
> > On Fri, 6 Apr 2018 at 11:03 pm, Tim Bain <tb...@alumni.duke.edu> wrote:
> >
> > > Are the topic subscriptions durable or non-durable? If the latter, were
> > the
> > > clients disconnected at the time the messages in question were sent?
> > >
> > > Also, do the subscribers in question use any selectors?
> > >
> > > Tim
> > >
> > > On Fri, Apr 6, 2018, 12:26 AM Lionel van den Berg <li...@gmail.com>
> > > wrote:
> > >
> > > > Another kind of vague one. We have found from our application logs at
> > > site
> > > > that some messages that we are sending appear to be sent OK on one
> end
> > > but
> > > > are never received by the subscriber.
> > > >
> > > > This seems to coincide with topics where the traffic volume is low,
> > does
> > > > anyone have any pointers on where to look first?
> > > >
> > >
> >
>

Re: Message appears to be sent but not received

[Tim Bain <tb...@alumni.duke.edu>]

I'd suggest you use a JMX viewer such as JConsole, and navigate through the
org.apache.activemq MBeans to find the subscription in question. On the
Attributes page under the subscription, you'll see counts for the number of
messages enqueued and dequeued.

If those don't match, it means your messages are not getting delivered to
the subscriber. (Could they be expiring before they can be consumed?)

If they match, but are lower than the number you sent to the topic while
the consumer was subscribed, we'll have to dig deeper.

Also, I understand from your response that these consumers have been
subscribed and connected the entire time the messages were being sent. Is
that what you're saying?

Tim

On Fri, Apr 6, 2018 at 7:11 AM, Lionel van den Berg <li...@gmail.com>
wrote:

> Non-durable an using the same connection as other topics that are still
> working. No filters.
>
>
> On Fri, 6 Apr 2018 at 11:03 pm, Tim Bain <tb...@alumni.duke.edu> wrote:
>
> > Are the topic subscriptions durable or non-durable? If the latter, were
> the
> > clients disconnected at the time the messages in question were sent?
> >
> > Also, do the subscribers in question use any selectors?
> >
> > Tim
> >
> > On Fri, Apr 6, 2018, 12:26 AM Lionel van den Berg <li...@gmail.com>
> > wrote:
> >
> > > Another kind of vague one. We have found from our application logs at
> > site
> > > that some messages that we are sending appear to be sent OK on one end
> > but
> > > are never received by the subscriber.
> > >
> > > This seems to coincide with topics where the traffic volume is low,
> does
> > > anyone have any pointers on where to look first?
> > >
> >
>

Re: Message appears to be sent but not received

[Lionel van den Berg <li...@gmail.com>]

Non-durable an using the same connection as other topics that are still
working. No filters.


On Fri, 6 Apr 2018 at 11:03 pm, Tim Bain <tb...@alumni.duke.edu> wrote:

> Are the topic subscriptions durable or non-durable? If the latter, were the
> clients disconnected at the time the messages in question were sent?
>
> Also, do the subscribers in question use any selectors?
>
> Tim
>
> On Fri, Apr 6, 2018, 12:26 AM Lionel van den Berg <li...@gmail.com>
> wrote:
>
> > Another kind of vague one. We have found from our application logs at
> site
> > that some messages that we are sending appear to be sent OK on one end
> but
> > are never received by the subscriber.
> >
> > This seems to coincide with topics where the traffic volume is low, does
> > anyone have any pointers on where to look first?
> >
>

Re: Message appears to be sent but not received

[Tim Bain <tb...@alumni.duke.edu>]

Are the topic subscriptions durable or non-durable? If the latter, were the
clients disconnected at the time the messages in question were sent?

Also, do the subscribers in question use any selectors?

Tim

On Fri, Apr 6, 2018, 12:26 AM Lionel van den Berg <li...@gmail.com> wrote:

> Another kind of vague one. We have found from our application logs at site
> that some messages that we are sending appear to be sent OK on one end but
> are never received by the subscriber.
>
> This seems to coincide with topics where the traffic volume is low, does
> anyone have any pointers on where to look first?
>