You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by BrSwPdx <bs...@pdxinc.com> on 2019/08/02 15:55:51 UTC
Odd issue with message consumers
We are seeing an odd issue with our embedded broker. We have our processors
registered as message listeners on a queue's consumer, but the consumer
appears to stop delivering messages via the OnMessage call. No errors are
thrown when this happens and we do not see any sort of dead lock or
contention in the thread dumps.
Adding the logging plugin shows that the queue is still active and receiving
messages, but they just do not seem to be consumed. This issue has us
stumped and I'm reaching out to see if anyone has encountered similar
behavior and found a cause/solution.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
Re: Odd issue with message consumers
Posted by BrSwPdx <bs...@pdxinc.com>.
Sorry about that, I've moved it to the user list.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Odd issue with message consumers
Posted by Justin Bertram <jb...@apache.org>.
No worries. Start a new thread on the user list with the details about the
issue you're seeing and I'll try to take a look.
Justin
On Fri, Aug 2, 2019 at 12:56 PM Maselbas, Dustin <DU...@dish.com>
wrote:
> Im sorry I was looking at a gradle build file before responding and
> transposed a version from another dependency, this is version 2.7.0. That
> aside please accept my apologies for my multiple infractions and thank you
> for your work and effort on this project.
>
> Dustin
>
> On 8/2/19, 11:47 AM, "Justin Bertram" <jb...@apache.org> wrote:
>
> This message originated outside of DISH and was sent by:
> dev-return-70854-DUSTIN.MASELBAS=dish.com@activemq.apache.org
>
> Couple of things:
>
> 1) There is no 1.2.7 release of ActiveMQ Artemis. The current
> release is
> 2.9.0, available here [1]. Older releases are available here [2]. As a
> matter of course I'd ask you reproduce your issue on the latest
> release.
> 2) As noted in my previous email, this is the "dev" list which is for
> developers who are working directly on the code-base of an ActiveMQ
> component. The "user" list is for users of ActiveMQ components who have
> questions, problems, etc. Please use the "user" list of these kinds of
> questions/issues. Links for the various mailing lists are available
> here
> [3].
> 3) Your issue doesn't appear related to the original. It's generally
> regarded as impolite to (try to) hijack other people's threads.
>
>
> Justin
>
> [1] http://activemq.apache.org/components/artemis/download/
> [2]
> http://activemq.apache.org/components/artemis/download/past_releases
> [3] http://activemq.apache.org/contact/#mailing
>
> On Fri, Aug 2, 2019 at 12:23 PM Maselbas, Dustin <
> DUSTIN.MASELBAS@dish.com>
> wrote:
>
> > Im also actively trouble shooting an issue on Artemis 1.2.7 where
> messages
> > being forwarded from one instance of Artemis via a bridge connection
> to
> > another will stop after exactly 1000 messages.
> >
> > AMQP -> ARTEMIS -> CORE -> ARTEMIS -> AMQP
> >
> > Right now It seems like it has to do with the MAX_DELIVERIES_IN_LOOP
> > constant in the QueueImpl class but I have not been able to pin it
> down
> > definitively. Profiling with YourKit is showing a possible thread
> deadlock
> > in the Epoll libs but disabling epoll on the bridge connection does
> not
> > rectify the issue.
> >
> > I have verified that we are using the most recent version of the
> Netty
> > Linux native module and have tested with Xms and Xmx set from 2GB to
> 12GB
> > with both G1GC and ParrallelGC to try and rule out any GC issues.
> >
> > We have also tested with persistence disabled to verify it was not a
> > journaling issue.
> >
> > Quick tests with 1.2.9 appear to have similar outcomes.
> >
> > Logs on the Artemis node pushing over the bridge to the other Artemis
> > instance constantly shows the "All the consumers were busy, giving
> up now"
> > debug message from line 2185 in QueueImpl.
> >
> > I was holding off on reporting anything until I had better
> documentation
> > but I figured this might be a good opportunity to bring it up.
> >
> > Dustin Maselbas
> > Principle Developer
> > Dish Networks
> >
> > On 8/2/19, 10:05 AM, "Justin Bertram" <jb...@apache.org> wrote:
> >
> > This message originated outside of DISH and was sent by:
> > dev-return-70852-DUSTIN.MASELBAS=dish.com@activemq.apache.org
> >
> > Are you using 5.x or Artemis?
> >
> > Also, the "dev" list (i.e. this list) is for developers who are
> working
> > directly on the code-base of an ActiveMQ component. The "user"
> list is
> > for
> > users of ActiveMQ components who have questions, problems, etc.
> >
> >
> > Justin
> >
> > On Fri, Aug 2, 2019 at 10:55 AM BrSwPdx <
> bsuwisutigasame@pdxinc.com>
> > wrote:
> >
> > > We are seeing an odd issue with our embedded broker. We have
> our
> > > processors
> > > registered as message listeners on a queue's consumer, but the
> > consumer
> > > appears to stop delivering messages via the OnMessage call. No
> > errors are
> > > thrown when this happens and we do not see any sort of dead
> lock or
> > > contention in the thread dumps.
> > >
> > > Adding the logging plugin shows that the queue is still active
> and
> > > receiving
> > > messages, but they just do not seem to be consumed. This
> issue has
> > us
> > > stumped and I'm reaching out to see if anyone has encountered
> similar
> > > behavior and found a cause/solution.
> > >
> > >
> > >
> > > --
> > > Sent from:
> > >
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
> > >
> >
> >
> >
>
>
>
Re: Odd issue with message consumers
Posted by "Maselbas, Dustin" <DU...@dish.com>.
Im sorry I was looking at a gradle build file before responding and transposed a version from another dependency, this is version 2.7.0. That aside please accept my apologies for my multiple infractions and thank you for your work and effort on this project.
Dustin
On 8/2/19, 11:47 AM, "Justin Bertram" <jb...@apache.org> wrote:
This message originated outside of DISH and was sent by: dev-return-70854-DUSTIN.MASELBAS=dish.com@activemq.apache.org
Couple of things:
1) There is no 1.2.7 release of ActiveMQ Artemis. The current release is
2.9.0, available here [1]. Older releases are available here [2]. As a
matter of course I'd ask you reproduce your issue on the latest release.
2) As noted in my previous email, this is the "dev" list which is for
developers who are working directly on the code-base of an ActiveMQ
component. The "user" list is for users of ActiveMQ components who have
questions, problems, etc. Please use the "user" list of these kinds of
questions/issues. Links for the various mailing lists are available here
[3].
3) Your issue doesn't appear related to the original. It's generally
regarded as impolite to (try to) hijack other people's threads.
Justin
[1] http://activemq.apache.org/components/artemis/download/
[2] http://activemq.apache.org/components/artemis/download/past_releases
[3] http://activemq.apache.org/contact/#mailing
On Fri, Aug 2, 2019 at 12:23 PM Maselbas, Dustin <DU...@dish.com>
wrote:
> Im also actively trouble shooting an issue on Artemis 1.2.7 where messages
> being forwarded from one instance of Artemis via a bridge connection to
> another will stop after exactly 1000 messages.
>
> AMQP -> ARTEMIS -> CORE -> ARTEMIS -> AMQP
>
> Right now It seems like it has to do with the MAX_DELIVERIES_IN_LOOP
> constant in the QueueImpl class but I have not been able to pin it down
> definitively. Profiling with YourKit is showing a possible thread deadlock
> in the Epoll libs but disabling epoll on the bridge connection does not
> rectify the issue.
>
> I have verified that we are using the most recent version of the Netty
> Linux native module and have tested with Xms and Xmx set from 2GB to 12GB
> with both G1GC and ParrallelGC to try and rule out any GC issues.
>
> We have also tested with persistence disabled to verify it was not a
> journaling issue.
>
> Quick tests with 1.2.9 appear to have similar outcomes.
>
> Logs on the Artemis node pushing over the bridge to the other Artemis
> instance constantly shows the "All the consumers were busy, giving up now"
> debug message from line 2185 in QueueImpl.
>
> I was holding off on reporting anything until I had better documentation
> but I figured this might be a good opportunity to bring it up.
>
> Dustin Maselbas
> Principle Developer
> Dish Networks
>
> On 8/2/19, 10:05 AM, "Justin Bertram" <jb...@apache.org> wrote:
>
> This message originated outside of DISH and was sent by:
> dev-return-70852-DUSTIN.MASELBAS=dish.com@activemq.apache.org
>
> Are you using 5.x or Artemis?
>
> Also, the "dev" list (i.e. this list) is for developers who are working
> directly on the code-base of an ActiveMQ component. The "user" list is
> for
> users of ActiveMQ components who have questions, problems, etc.
>
>
> Justin
>
> On Fri, Aug 2, 2019 at 10:55 AM BrSwPdx <bs...@pdxinc.com>
> wrote:
>
> > We are seeing an odd issue with our embedded broker. We have our
> > processors
> > registered as message listeners on a queue's consumer, but the
> consumer
> > appears to stop delivering messages via the OnMessage call. No
> errors are
> > thrown when this happens and we do not see any sort of dead lock or
> > contention in the thread dumps.
> >
> > Adding the logging plugin shows that the queue is still active and
> > receiving
> > messages, but they just do not seem to be consumed. This issue has
> us
> > stumped and I'm reaching out to see if anyone has encountered similar
> > behavior and found a cause/solution.
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
> >
>
>
>
Re: Odd issue with message consumers
Posted by Justin Bertram <jb...@apache.org>.
Couple of things:
1) There is no 1.2.7 release of ActiveMQ Artemis. The current release is
2.9.0, available here [1]. Older releases are available here [2]. As a
matter of course I'd ask you reproduce your issue on the latest release.
2) As noted in my previous email, this is the "dev" list which is for
developers who are working directly on the code-base of an ActiveMQ
component. The "user" list is for users of ActiveMQ components who have
questions, problems, etc. Please use the "user" list of these kinds of
questions/issues. Links for the various mailing lists are available here
[3].
3) Your issue doesn't appear related to the original. It's generally
regarded as impolite to (try to) hijack other people's threads.
Justin
[1] http://activemq.apache.org/components/artemis/download/
[2] http://activemq.apache.org/components/artemis/download/past_releases
[3] http://activemq.apache.org/contact/#mailing
On Fri, Aug 2, 2019 at 12:23 PM Maselbas, Dustin <DU...@dish.com>
wrote:
> Im also actively trouble shooting an issue on Artemis 1.2.7 where messages
> being forwarded from one instance of Artemis via a bridge connection to
> another will stop after exactly 1000 messages.
>
> AMQP -> ARTEMIS -> CORE -> ARTEMIS -> AMQP
>
> Right now It seems like it has to do with the MAX_DELIVERIES_IN_LOOP
> constant in the QueueImpl class but I have not been able to pin it down
> definitively. Profiling with YourKit is showing a possible thread deadlock
> in the Epoll libs but disabling epoll on the bridge connection does not
> rectify the issue.
>
> I have verified that we are using the most recent version of the Netty
> Linux native module and have tested with Xms and Xmx set from 2GB to 12GB
> with both G1GC and ParrallelGC to try and rule out any GC issues.
>
> We have also tested with persistence disabled to verify it was not a
> journaling issue.
>
> Quick tests with 1.2.9 appear to have similar outcomes.
>
> Logs on the Artemis node pushing over the bridge to the other Artemis
> instance constantly shows the "All the consumers were busy, giving up now"
> debug message from line 2185 in QueueImpl.
>
> I was holding off on reporting anything until I had better documentation
> but I figured this might be a good opportunity to bring it up.
>
> Dustin Maselbas
> Principle Developer
> Dish Networks
>
> On 8/2/19, 10:05 AM, "Justin Bertram" <jb...@apache.org> wrote:
>
> This message originated outside of DISH and was sent by:
> dev-return-70852-DUSTIN.MASELBAS=dish.com@activemq.apache.org
>
> Are you using 5.x or Artemis?
>
> Also, the "dev" list (i.e. this list) is for developers who are working
> directly on the code-base of an ActiveMQ component. The "user" list is
> for
> users of ActiveMQ components who have questions, problems, etc.
>
>
> Justin
>
> On Fri, Aug 2, 2019 at 10:55 AM BrSwPdx <bs...@pdxinc.com>
> wrote:
>
> > We are seeing an odd issue with our embedded broker. We have our
> > processors
> > registered as message listeners on a queue's consumer, but the
> consumer
> > appears to stop delivering messages via the OnMessage call. No
> errors are
> > thrown when this happens and we do not see any sort of dead lock or
> > contention in the thread dumps.
> >
> > Adding the logging plugin shows that the queue is still active and
> > receiving
> > messages, but they just do not seem to be consumed. This issue has
> us
> > stumped and I'm reaching out to see if anyone has encountered similar
> > behavior and found a cause/solution.
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
> >
>
>
>
Re: Odd issue with message consumers
Posted by "Maselbas, Dustin" <DU...@dish.com>.
Im also actively trouble shooting an issue on Artemis 1.2.7 where messages being forwarded from one instance of Artemis via a bridge connection to another will stop after exactly 1000 messages.
AMQP -> ARTEMIS -> CORE -> ARTEMIS -> AMQP
Right now It seems like it has to do with the MAX_DELIVERIES_IN_LOOP constant in the QueueImpl class but I have not been able to pin it down definitively. Profiling with YourKit is showing a possible thread deadlock in the Epoll libs but disabling epoll on the bridge connection does not rectify the issue.
I have verified that we are using the most recent version of the Netty Linux native module and have tested with Xms and Xmx set from 2GB to 12GB with both G1GC and ParrallelGC to try and rule out any GC issues.
We have also tested with persistence disabled to verify it was not a journaling issue.
Quick tests with 1.2.9 appear to have similar outcomes.
Logs on the Artemis node pushing over the bridge to the other Artemis instance constantly shows the "All the consumers were busy, giving up now" debug message from line 2185 in QueueImpl.
I was holding off on reporting anything until I had better documentation but I figured this might be a good opportunity to bring it up.
Dustin Maselbas
Principle Developer
Dish Networks
On 8/2/19, 10:05 AM, "Justin Bertram" <jb...@apache.org> wrote:
This message originated outside of DISH and was sent by: dev-return-70852-DUSTIN.MASELBAS=dish.com@activemq.apache.org
Are you using 5.x or Artemis?
Also, the "dev" list (i.e. this list) is for developers who are working
directly on the code-base of an ActiveMQ component. The "user" list is for
users of ActiveMQ components who have questions, problems, etc.
Justin
On Fri, Aug 2, 2019 at 10:55 AM BrSwPdx <bs...@pdxinc.com> wrote:
> We are seeing an odd issue with our embedded broker. We have our
> processors
> registered as message listeners on a queue's consumer, but the consumer
> appears to stop delivering messages via the OnMessage call. No errors are
> thrown when this happens and we do not see any sort of dead lock or
> contention in the thread dumps.
>
> Adding the logging plugin shows that the queue is still active and
> receiving
> messages, but they just do not seem to be consumed. This issue has us
> stumped and I'm reaching out to see if anyone has encountered similar
> behavior and found a cause/solution.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
>
Re: Odd issue with message consumers
Posted by Justin Bertram <jb...@apache.org>.
Are you using 5.x or Artemis?
Also, the "dev" list (i.e. this list) is for developers who are working
directly on the code-base of an ActiveMQ component. The "user" list is for
users of ActiveMQ components who have questions, problems, etc.
Justin
On Fri, Aug 2, 2019 at 10:55 AM BrSwPdx <bs...@pdxinc.com> wrote:
> We are seeing an odd issue with our embedded broker. We have our
> processors
> registered as message listeners on a queue's consumer, but the consumer
> appears to stop delivering messages via the OnMessage call. No errors are
> thrown when this happens and we do not see any sort of dead lock or
> contention in the thread dumps.
>
> Adding the logging plugin shows that the queue is still active and
> receiving
> messages, but they just do not seem to be consumed. This issue has us
> stumped and I'm reaching out to see if anyone has encountered similar
> behavior and found a cause/solution.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
>